Photo uploads failed for 112 minutes on September 21

This should not have happened. On September 21, for 1 hour and 52 minutes, photos taken with Timemark could not be uploaded to Teamspace. Worse, the failure was silent: the app treated those uploads as successful, so it never retried them. About 39,000 photos never reached their teams, and they will not arrive on their own.

For a small number of people, the consequence is worse than a delay: their photos are gone permanently. We explain who, and why, below.

We are sorry. For people whose work depends on getting site photos to their team the same day, this is not a small thing.

What you need to do

For almost everyone, your photos are still on your phone, in your local album. They will not sync by themselves.

Open Teamspace, check your photos from that window, and upload again anything that is missing.

If you use "Save photos to Teamspace only"

There is one situation where photos are gone for good, and we need to be direct about it.

That setting, which a team admin can turn on for the whole team or a member can turn on individually, keeps photos out of your phone's camera roll to save storage. Photos are held in the app's private storage and deleted once they have been uploaded to Teamspace.

During the incident our servers returned a success status for uploads that had actually failed, and the app deleted the local copy on the strength of that status alone. Those photos cannot be recovered.

We want to be clear about where the fault sits. This is a design failure on our side, not simply a server outage. A local copy should never be deleted because a response said "success"; it should be deleted only after the photo is confirmed stored. And when an upload cannot be confirmed, the photo should fall back to your camera roll rather than disappear. Neither safeguard existed, and that is why a two-hour outage turned into permanent loss for some people.

From our data, at most around 330 people took photos during the outage window without a copy going to their device album. That is 0.8% of everyone affected, and the true number is lower, because some of those uploads did succeed. We cannot tell you precisely who, and we are not going to pretend otherwise.

If this happened to you or your team, write to support@timemark.com. We are handling these case by case.

Who was affected

43,348 people tried to upload photos during the incident window. About 39,000 photos failed and did not retry.

The window in UTC:

Local times differ by country. For Indonesia, Vietnam and Thailand this was late afternoon, 16:46 to 18:38. For the Philippines, Malaysia and Singapore, 17:46 to 19:38. For Brazil, early morning, 06:46 to 08:38.

What was not affected

• For 99.2% of affected users, no photos were lost. They are still in the local album on the device that took them. The exception is the group described above.

• Watermarks, timestamps and GPS data are intact. Photos taken during the outage carry the same verifiable information as always.

• Already uploaded photos were never at risk. Nothing that had reached Teamspace before the incident was affected.

• The camera itself kept working. Taking photos, applying watermarks and saving locally all worked normally throughout.

What happened

We shipped an update that added real-time notification support for iOS. That update reads and writes a table holding roughly two million device records, and it runs every time a user signs in or signs out.

We forgot to add one database index to production. Without it, every one of those writes scanned the entire table. Each scan held a database connection far longer than it should have. Within half an hour the connections were exhausted, and any request that needed the database, including photo uploads and opening Teamspace, began to fail.

The silent-failure part was a separate mistake. When the team service could not answer, it returned a generic error that our upload service interpreted as success. The app therefore marked those uploads as done and never queued them for retry. This is why the photos did not come back on their own, and it is the failure we regret most.

Timeline (UTC)

Why it took 112 minutes

Three things made this slower than it should have been.

Our monitoring told us the problem had gone away. The slow-query alert fired at 09:46. Minutes later the monitoring system sent an automatic all-clear: the alert had resolved itself. The team was in a meeting, and that all-clear was enough to close the question. Nobody opened a console to check whether the underlying condition was actually gone. It was not. Thirty minutes passed before anyone looked again.

An automatic all-clear is not evidence that a problem is fixed. It only means the symptom stopped tripping a threshold. We built a system that could tell us to stop worrying, and we believed it.

The rollback was incomplete. At 10:37 we had rolled back the production services but not the canary environment, which was still sending the same harmful traffic to the same database. We did not realise this until much later, so the first rollback looked like it had failed and sent us searching in the wrong direction.

We could not see into the database. By the time we suspected connection exhaustion, the database console itself was unreachable because it needed a connection too. Diagnosis had to proceed by elimination, service by service, which is what consumed most of the second hour.

What we are changing

The list below is not a set of patches. It follows from four principles we are now designing the product around.

1. Photo authenticity is never affected

Timestamps and addresses come from our servers, not from your phone's clock or settings. That is deliberate. A timestamp the user can change is not evidence, and the whole point of a Timemark photo is that someone else can trust it.

That service runs separately from Teamspace, on its own infrastructure. It was not touched by this incident, and by design it does not share a failure path with photo uploads. It also caches, so it keeps working through short periods without a connection.

The watermark itself is composed and written onto the image on your device. Every photo taken on September 21, including the ones that failed to upload, carries the same verifiable time, location and watermark as any other photo.

2. Field work does not depend on our servers

Selecting a project, applying a team watermark, filling in a checklist, clocking in and out: all of these must keep working offline, and they must keep working when our servers are down.

Someone standing on a site with no signal, and someone standing on a site while we are having a bad day, should not be able to tell the difference.

3. Photos are never lost

Every photo that cannot be uploaded goes into a pending queue on the device. When network and service return, the queue drains on its own. No action needed from you.

This is exactly what failed on September 21. The upload was marked successful, so nothing entered the queue, so nothing retried. We are rebuilding that path, and adding an upload status list in the app so you can see the queue yourself rather than trusting that it worked.

No local copy will ever be deleted again without verified confirmation that the photo is stored on our side. A generic success response is not proof.

And when an upload cannot be confirmed, the photo falls back to your camera roll rather than sitting in app storage alone. Saving storage space must never cost you the photo. These two changes together are what would have prevented the permanent losses described above.

4. Recovery is fast

One-click rollback across every environment at once, including staging traffic. Restart and return to the last known good version in minutes, not in an hour.

Supporting work

• An automatic all-clear no longer closes an investigation. A resolved alert means a threshold stopped tripping, not that the cause is gone. Recovery notices now require a human check before the incident is closed.

• Alert hygiene. Every noisy non-urgent alert is being removed, so that the alerts left are ones we act on immediately.

• Database schema in automated code review, with index coverage and slow-query risk checked before any change reaches production.

• Maximum query execution time enforced at the database level, so one slow query can no longer hold a connection indefinitely.

• Separating the databases behind supporting features from those behind core features. A notification feature must never be able to take down photo uploads.

• A full audit of error codes across server, app and web, so a failure in one service can no longer be read as success by another. This is the specific gap that cost you 39,000 photos.

• Staged rollout enforced. No deployment goes straight to production without observation time in staging first.

If this caused you a problem

If this incident caused you a problem we have not accounted for, or if you lost photos you cannot recover, write to support@timemark.com and we will look at your case individually.