Discussion: periodic API failures

Fairly often, I get a call failure with the error "server_error".  Most recently I saw this from locations.streetInfo.

Is there some further way to figure out what's going on or is the API just flaky?

Should I retry when this happens or is it likely to be persistent?
I saw this when calling the API without waiting to not pass the 3 calls per 3 seconds. but after changing the code to dont call ever more than 3 times... it didnt happen again.
I think they changed the way the limit works in the last two weeks.. but I'm not sure...
I changed my code to sleep for 1 second and then retry the call (once) when this happens and that seems to have helped.  I still don't know what causes the server_error, but it seems to be a transient problem.

Lately, I've noticed HTTP failures on the GET requests to the API.  The sleep/retry seems to help this as well.  I'm not sure how the API server cluster is set up, but there do seem to be quite a number of transient failures.
The rate limiter should return a 503 error if you're over the limit.

If you reproduce this error in the future, could you please reply to this thread with the full response?
Last error I got was a 500 at 3am this morning (Pacific time).  I don't have the full response (if there was one). In case it helps, PHP printed this:

Warning: file_get_contents(http://api.glitch.com/simple/locations.streetInfo?street_tsid=LA5146Q41J528U9): failed to open stream: HTTP request failed! HTTP/1.1 500 Internal Server Error

Note that I waited one second and retried the request, which failed with the same 500 error (after which my script aborted).
Occasionally, the API also returns an error code itself (JSON data with the error set):
API "http://api.glitch.com/simple/locations.getStreets?hub_id=89" returned error misc_fatal.
This happened today at 17:30 Pacific.
So, the API has continued to be flaky, but today was the first time I saw the error hub_not_found come up (usually it's misc_fatal or a protocol error 500).

API "http://api.glitch.com/simple/locations.getStreets?hub_id=123" returned error hub_not_found. Unable to get API data for region ID 123!

This occurred at 5:30pm Pacific on July 26.
Now I've seen about half a dozen misc_fatal errors in the last 18 hours or so. Any chance that we can get some dev response as to what causes these and whether there is a work-around?
I get those too, sometimes. My script just record the error in a log then move on to the next player. The script pulls in IDs of 40 players that need updating at once, and since each player takes ~3 seconds, the same call will be made again in around 2 minutes, by which time the error is usually resolved.

The real oddity is that sometimes, player's POL street tsid actually changes - that is, I get street not found errors. The first few times this happened the players were developers - annoying, but not entirely unexpected. Then, after new players start coming back, I noticed this happening to normal players too. These are new players, or players who haven't played since the last beta. I'm not sure what's happening, but this happens on average about once a day, and is quite annoying. Is this a bug, or something we should expect to see once in a while? (The last one I observed is player PIFURLF7VO53GLO, whose home street ID changed from LHVGFUBK0P538RL to LHVOLN2GB1638O6)
I think the answer is: if there is an error or if the content is empty... retry!  I retry 3 times if something strange happens.

and I never call more than 3 times per 3300ms. but anyway some times it takes 3 seconds to get the data :(

about the home street id changing... ouch! that's a problem. I'm not saving history so I have no idea. maybe if someone "extends" the HS... it creates a new street, and the id changes?
I sleep for 1s then retry on failure, but if the retry also fails I abort processing until the next pass.

However, I don't believe this is due to API throttling since the most common error I get is misc_fatal and the same thing appears to happen in the game (the game will freeze for 2-5s). At least I don't think throttling would be applied to the game client, but maybe so...

Since the server which accesses the API and my home machine which accesses the game are entirely different locations, I suspect that TS needs to add more servers, beef up the hardware or improve the performance of the API and game servers.
The home street TSID shouldn't change, but it might change for those who were still on the old beta new housing test and didn't log on until recently, and then after they log on there's a new TSID since they now get the final release house.
Another big batch of failures in the last 12 hours, this time with error "server_error".

Knock, knock; anyone home at TS?