unstable

The last evenings the chat has been very unstable. I changed yesterday to US server, but today  the us server also has been very unstable. 

So I changed back to europe for this evening. I don't know any other way to report this than here. It is really annoying and becomes the most imprtant topic on the chat. Wrong focus..

i'm not showing any connection or load issues with either server, can you be a bit more clear on what you mean by "unstable"?

OK - all my users  included me, are throwm out all the time. They are just disconnected. Need to refresh to see whats happening..

The europe server is stable now for us.. has been for the last 15 minutes... 

I had issues too on Saturday for the US server as I was disconnected via Android.

after decreasing the fault tolerance for server notifications from 95% memory utilization for greater than 5 minutes, i set it for anytime the usage goes above 80% for any length of time. there appears to be memory "spikes" that happen that appears to be followed by a mass disconnects. I'm trying to track down the cause of it, it is happen to both servers and NA has significantly more ram than EU has. I'll be going over line per line of the babble.js tomorrow and redoing somethings in it, I think i'm simply trying to store too much in memory for the online lists. 

Changed Status from Pending to Work in Progress

I'm also gonna add in a delay for the "disconnect" screen, so if its just a quick disconnect/reconnect, it wont show. cause i was connected to my server here with my android phone (with the data connection, trying to burn up more LTE before i pay my bill this friday, they want to get rid of the good cheap plan forcing me to go to a higher plan, i'm gonna use every megabyte!).  i noticed a few "disconnects" with quick reconnects, that i could see being annoying. 

The europe server was extremely unstable now. Every minute I was thrown out. Changed over to US now. I give some of my moderators access to scp just so they can alter server

i've updated the service on both servers and have restarted them. NA looks to be using half the ram it was before, and so is Europe. 

1 hour ago, Kjell Iver Johansen said:

The europe server was extremely unstable now. Every minute I was thrown out. Changed over to US now. I give some of my moderators access to scp just so they can alter server

 

i think part of the problem is there no delay with the "disconnect" screen it fires as soon as the connection is lost. if the connection is client side that is an immediate thing (like your net drops out).  it takes 30 seconds or there about, for the client to realize that server has disconnected them, so i don't think all them are server related. with the few disconnects i noticed last night, i was only able to confirm the one was with the ram spike usage. the others seemed random. when inspecting my phones data connection logs, they seem to correspond to data connection interruptions. something you don't always "see" but like i said when its a client side disconnect its an immediate thing. so in the next version, 2.6.3, I'll be including a delay on it. 

The NA server appears to be working well, however the europe server has issues. I am not sure why, that server hasn't been alright since day one and i think that it is large part due to the datacenter it is in. i created another one in that datacenter and had similar issues.  I might have to take it offline for now and move it to another datacenter. 

ok.. I will not use it then. seems like the chat is oki now - thanks

checking their support page and downtimes, they seem to have a lot of high load that effects the droplets and connectivity issues there. I've sent an email to their support, and checking out the other 2 datacenters in europe.

I've had a group in chat this evening and about 15 min  we witnessed more babble disconnect and members being kicked out.

9:12PM EST disconnected from babble.  

Seems to be happening quite a bit right now.  refresh doesn't work, having to log out and back in the babble chat room

I've been tracking this issue for the last week, there seems to be a memory leak in the node script for babble that is causing node to crash and forever to restart itself. I have reworked the babble.js and will be releasing a new version later tonight or tomorrow, that from my testing seems to resolve the memory leak, hopefully fixing the connection issues. so be on the look out for 2.6.3

Changed Status from Work in Progress to Waiting

Changed Fixed In to Next Version

That is great News . Last saturday evening the us server was throwing us out every minute. We then changed to EU and since that all OK. Will check out new version

its been released and both chat service regions have been update to work with this new version, previous versions will not function with the chat service due to the changes and addressing the security issue. 

I'm showing huge memory spikes on the europe server still, and it only has 50 connected users atm and it is currently eating 6 times the ram than the NA server which has 380 users connected...so i will more than likely be taking that server offline, there is no reason why it should continue to be running with so much resources. 

whoa european server just shot up to 18k users and then slowly ticked back down to around 50 after about 30 seconds...that is interesting...not sure what has caused that. 

Strange. I just checked my /stats, and it shows 48 users online at the most. Is that concurrent logged in users?

Be aware that there something else fishy with the stats. Even though most of them was active last evening the date is wrong. Might be something with the servers? I notice myselve are correct because I wrote in the chat now on the latest release

2 minutes ago, Kjell Iver Johansen said:

Strange. I just checked my /stats, and it shows 48 users online at the most. Is that concurrent logged in users?

Be aware that there something else fishy with the stats. Even though most of them was active last evening the date is wrong. Might be something with the servers? I notice myselve are correct because I wrote in the chat now on the latest release

none of that goes thru the chat service, that is all done on your server. stats might be skewed, they were in the section of code where the bots hang out and the bots weren't working, but that was only in 2.6.0, not sure why the most recent is showing over a month old. i'll check out the stats. i think something is seriously wrong with the european host, its setup identical to the NA server, but its eating up about 6 times the ram with about 6 times less users connected. 

1 minute ago, Michael said:

i think something is seriously wrong with the european host, its setup identical to the NA server, but its eating up about 6 times the ram with about 6 times less users connected. 

I've been changing back and forth, and that might be the reason the stats are broken. Anyway no problem. I have no problem if you close the Eu server, but personally I like the ability to change server in case a server is down for whatever reason, so an alternative could be great.

1 minute ago, Kjell Iver Johansen said:

I've been changing back and forth, and that might be the reason the stats are broken. Anyway no problem. I have no problem if you close the Eu server, but personally I like the ability to change server in case a server is down for whatever reason, so an alternative could be great.

i'll probably end up redoing it in another datacenter, maybe frankfurt or london. i've had issues when creating other instances in the amsterdam datacenter. I'm thinking of purchasing the chat service its own domain, right now i'm just using my auxiliary url's like codingjungle.net and provisionists.net

just had a huge spike of online users for NA, it went from about 350 to over 8k in a few seconds, then about 10 seconds later it went down to around 3700 and now its hovering at about 1100.  i'm trying to figure out now are these legit connections. I might have to remove the menu counts on the chat service and rethink how how i do it. 

i think it might be outdated chat clients trying to connect, not sure yet.