The story about the restriction and outage of LightWidget

In June 2024, we experienced one of the most prolonged downtimes for our Instagram widgets in the ten years since we began providing our services. Here is the story of what has happened.

First and foremost, we sincerely apologize to all of our users for the inconvenience caused by the problems with your widgets. We understand how frustrating this was and acknowledge that we let you down.

This post serves as a “post-mortem” of the entire incident, explaining what happened, how we solved the problem, the lessons we learned, and the improvements we made.

How LightWidget works

We can provide our services only thanks to the official Instagram APIs provided by Meta. These APIs are necessary to legally obtain the required information and convert them into widgets.

Our service is entirely dependent on Meta. Any issues or downtime on Meta’s end directly affect our services. We can only fix or handle issues that occur on our end. We cannot fix problems arising on Meta’s end and must wait for Meta’s team to resolve them.

An unusual situation on Meta’s end caused our downtime. This issue was not related to our codebase. Meta restricted our access to the API due to an alleged failure to respond to their messages on time. Spoiler alert: we always reply to all their messages promptly.

How developing the apps using Meta’s API works

If you have never had a chance to use one of Meta’s API products, like Instagram API, you must know that it is not a walk in the park.

First, you must create an app on the Meta for Developers platform. After that, you need to develop a functional application and go through several reviews, including Business Verification, App Review, Data Use Checkup, Data Protection Assessment, Tech Provider Verification, and more.

Once your App is in “Live” mode, periodic reviews are required to keep it operational. While this process can be cumbersome for developers, it is essential to maintain high-quality services and prevent the misuse of Meta APIs.

What has happened

Here is the breakthrough day by day of what has happened.

How developing the apps using Meta’s API works

If you have never had a chance to use one of Meta’s API products, like Instagram API, you must know that it is not a walk in the park. First, you must create an App on the developer platform, build an app, and then go through many reviews, like Business Verification, App Review, and Data Use Checkup. There are also periodic reviews now and then that you must complete to have your app up and running.

As it’s annoying to the developers who build apps, it is necessary to have high-quality services that do not misuse the APIs.

May 23

On May 23, we received an email from Meta stating they identified the problem on our website during a regular review. The email said that Facebook Login is Broken. We use Facebook Login behind Business Connection.

The functionality itself was working fine. Numerous users connected their accounts using Business Connection on that day and the following days, so we knew it was a reviewer error. It had happened before, and we knew that we must clarify the exact steps to proceed when we will send a reply.

In most cases like this, the reviewer uses a Facebook user account without an Instagram account. As you know, we create Instagram widgets, so having an Instagram account is essential for testing our App.

When we received the email, most of our small team was on vacation. We had a quick call and decided to handle it once the team was in place and rested. We had to act by June 24, which gave us plenty of time to respond.

June 7

That day, we prepared a screencast and exact testing instructions and sent them to the reviewer. We didn’t make any changes to our code regarding Facebook Login.

1,5 hours later, we received a response stating that the review went correctly:

We have reviewed your response to a violation for LightWidget, as described below, and determined this app meets the requirements of our Platform Terms and Developer Policies.

Props to the Meta team for the fast review! We were all set.

June 11

At 22:05, we received another email stating, “LightWidget is restricted due to violations”. The reason was “Timely response to our requests“. We were genuinely shocked.

We quickly looked at the details and knew something was not OK on Meta’s end. The details of the restriction stated that:

To restore the app, respond to the open violations listed below:
Resolve violation, Resolved Jun 7, 2024

First, we responded and received a successful review. Even the message about the restriction in the Meta developers panel has the status “Resolved.”

Second, we quickly checked all the messages in our email and developers panel to see if we still needed to respond to anything. All was good; we didn’t miss any messages or alerts.

There was still an option to resubmit the form for the violation we received on May 23. We updated our response, asking what happened and why we got the restriction. We hit the Submit button, and…
…the Meta server returned a critical 500 error and could not process our request.

There is a separate panel to send the appeal to if your App is restricted. We wanted to use it but without any luck. The form stated:

None of your apps are restricted.

So, that was not a viable option for us either, even though our app is restricted.

The obvious way would be to send a bug report. However, in the past few months, sending the bug reports has been very limited, and in most cases, you get a default reply:

Advanced technical support via this channel is no longer publicly available for this product. Instead, these issues will be supported via the Developer Community forum. We believe engagement through the community will aid collective understanding of our products and we are continuously improving the experience of the forum. We will also monitor the forum to support developers on critical issues facing their business.

To us, the developer community is like an old-fashioned forum where you can ask questions and count on a fellow developer to give you some reply. Although the idea seems excellent, when there is a bug on Meta’s end, you need access to the code, logs, etc., of Meta’s products. The “regular” user of the forum cannot fix such an issue. One of the Meta developers would have to read the community post added by a developer and report the issue. However – do they read it often? We don’t know.

Nevertheless, we quickly scanned the Community forum and discovered others were facing a similar issue. At least five other developers had the same problem. Meta restricted their apps despite all resolved violations without any way to reply/appeal.

June 12

Here is where we started to think out of the box. Since we can’t submit a bug report and have no way to appeal or reply to the restriction, we decided to start a new App Review. We requested new permission for LightWidget, which we don’t really need. That way, we could reach a real reviewer who works at Meta and point out a problem on their platform.

We created a screencast showing the problem and the inability to proceed. We also wrote this in the submission details.

Later that day, we received an email that our app review was canceled. There was no additional feedback.

Did someone check the problem with the restrictions we and others face? We have no idea.

In the meantime, new reports from other developers facing the same issue, totaling 15-16 cases like this, emerged.

June 13

In the morning, nothing changed, apart from the number of affected developers – 32 upvotes on the Community thread.

Since there was no official reply or any movement from the Meta team, we decided to create a new app to replace the restricted version. The first obstacle we faced was that we needed to make two apps instead of one. Why? Meta decided to disallow new apps from accessing the Basic Display API (behind the Consumer connection on our website) and the Graph API (behind the Business connection). It is not possible to add these two APIs to the same App. The restricted LightWidget app version has access to both APIs. Still, we created it many years ago, so some things changed along the way.

We prepared the test servers, wrote submissions, recorded the screencasts, and submitted the apps for review.

In the submission and the screencast, we added information about the current situation regarding our restricted application. We added the links to the Facebook Community Post and showed the problem we and others face. We asked if the reviewer has any authority or possibility to report it somewhere so someone can fix the issue.

The downside is that the review will take approximately five to ten days, and there is no guarantee that they will approve it on the first try.

June 14

Despite the growing number of users (44) experiencing the same issue, light might be at the end of the tunnel. Two users reported that Meta resolved the “Timely response” violation, and their apps are no longer restricted. Unfortunately, this is not the case for LightWidget and other affected apps.

In the afternoon, Meta emailed us the results of the app review for our new application, LightWidget for consumers. We created this app App to fix the consumer connection on our page. Fortunately, Meta reviewers approved the App during the first review!

We were thrilled, as this would partially address the problem. We switched the new version to Live mode from development mode. We began writing the code to use the unrestricted version on our production website.

The disappointing part is that the reviewer should have responded to our message about the issue with our restricted App. We are confident that the reviewer must have seen the message in the video and the text details. Once they reviewed the App, there were no follow-ups or ways to contact the reviewer. Ultimately, this method of notifying Meta about the problem was a dead end.

Later that day, to our surprise, we received another email about the newly approved App: Complete Data Use Checkup for LightWidget for consumers.

Our immediate reaction was: What is going on?!?!?! Each App must complete a Data Use Checkup annually for those unfamiliar with it. It is very similar to the App Review and involves almost identical questions. The Data Use Checkup is typically required a year after the App goes live, not two hours after. Something is seriously wrong with the Meta Developer Platform.

The email stated we had six days to complete the checkup. Otherwise, our new App will be restricted. We promptly submitted all the required information again.

June 15

The post about the restriction is now the most up-voted post on the entire Developers Community Forum, yet Meta has neither responded to it nor fixed the issue.

We spent most of the day preparing our service to integrate the replacement app. We decided to code it in a way that allows us to utilize both apps—the replacement and the restricted version. The main reason was that if Meta magically removed the restriction, all existing widgets should begin working without user interaction.

We have thousands of Instagram accounts connected to our App, some of which were linked even ten years ago when we first launched our service. Since then, Instagram/Facebook/Meta made dozens of changes to Instagram APIs. There were deprecations of the old API, split between two separate APIs, user ID changes, and many more. If someone built the widget, let’s say, four years ago and decided to renew the token, there would be at least 2 data migrations we would need to perform on the Instagram account user data so the user wouldn’t lose his widgets. On top of that, we must now add another layer that will utilize the new App and the migration.

Luckily, our code has hundreds of various tests. These helped us cover most of the scenarios without breaking previously created migrations. It took a bit longer than we anticipated, but we finally managed to handle it.

In the meantime, we discovered that the new App’s Data Use Checkup was complete, so we should be ready to release the fix tomorrow.

June 16

The good news is that we have successfully released a fix for the consumer connection. We identified a minor bug, applied the necessary fixes, conducted tests, and deployed the update to all users.

The bad news is that the restriction remains for the old App. As a result, the only way to fix widgets using the business connection is to temporarily switch to the consumer connection.

We have decided to postpone sending expired token notifications until tomorrow. This delay will help us catch any remaining bugs or issues that might arise when multiple users renew their tokens. Our primary goal is to refrain from sending notifications that could result in broken pages or other problems, further burdening our support. Our support ticket queue already contains hundreds of questions, and we want to minimize the additional load.

June 17

The day prior, we decided not to send the expired token notifications. It was a good decision because many people reported that the Instagram website throws a 400 Session Invalid error, preventing the connection of new accounts and refreshing the tokens. Well, it is yet another issue on Meta’s end that we can’t fix:/

We continued to reply to the support tickets you guys submitted. Since this problem affects most widgets, we received nearly 800 support tickets until June 17. It will take some time to reply to all of them. We apologize if you do not receive a response within 24 hours. It is difficult for us to reply to all your questions simultaneously, fighting to restore the full functionality of our service. We promise to respond to all support tickets!

Due to the lack of other ideas, apart from waiting for something to happen with the restriction, we did some research and found a guy on LinkedIn who had helped us with one of the support tickets in the past (when it was still possible to submit bug reports to Meta). We decided to reach out to him on LinkedIn. He didn’t respond, but if you are reading this by any chance, we are sorry for bothering you. It was an act of desperation.

We also decided to turn off the business connection on our website and replace the buttons with information about what is going on. This move prevented the Facebook website from displaying the message that the App was inactive without further explanation.

June 18

Something is finally clearing up. At least, that is what we think. On June 17, we removed the business connection buttons, which likely triggered a testing script on Facebook, resulting in another “Facebook Login is broken” violation.

This time, the violation was valid because the Facebook Login button was no longer present, preventing Meta from testing the integration properly.

We reverted the changes, restored the button, and prepared another submission. We also mentioned the trouble with the restriction and requested that Meta escalate the issue developers are facing. We hope Meta will remove the API restriction if they resolve the violation.

After two hours, we received a response to our submission. Meta decided that the “Facebook Login is broken” violation had been successfully addressed. However, the restriction was still in place.

We noticed that Meta updated the restriction description on their dashboard. They added the following sentence:

If you are receiving this violation and have already responded to all other violations, your responses are in process.

Previously, it only showed a message that we must resolve the already resolved violations. Someone on Meta’s end may have noticed that the previous restriction message was unclear, leading many users to wonder what was happening and why the App was still inactive despite all resolved violations.

You need to read it carefully to catch that. If you received the restriction before the update, you might easily overlook the new addition to the restriction description. It could explain why Meta lifted some restrictions for specific users but not for others. There must be a queue where reviewers check apps one by one. All we can do now is wait for our turn.

June 19 – June 24

Little progress was made regarding the restriction between these dates. We continuously monitored the Community Forum, our App Dashboard, and our email for any updates or changes concerning the restriction, but unfortunately, there were no developments.

In the meantime, we worked diligently to respond to as many support tickets as possible. We also developed some internal tools to help us identify issues based on your support tickets, enabling us to respond faster.

Despite our efforts, hundreds of support tickets are still in our backlog. We know it will take a few more days to address all of them.

June 25

We received the first result of the App Review for the new App intended to replace the Business connection. Unfortunately, the reviewer rejected it for an unexpected reason:

We were not able to approve the permission that you provided because there’s no Instagram Business account mention. As a result, we cannot completely verify the permission that was requested.

We included the testing credentials for the Business Instagram account, labeling it as “Credentials for the Instagram test account.”

It seems that the reviewer (or an AI/automated system) did not correctly identify the account’s type. Apparently, if the submission details do not explicitly include the phrase “Instagram Business account,” the review is automatically rejected.

We updated the label from “test account” to “Instagram Business account” and resubmitted the review to address the issue.

June 26

We received the response to the submission that we sent yesterday:

We have determined that your app’s use case for the requested permission is valid. However, we were unable to replicate the experience of how the permission will be used within your app. Specifically, the Facebook test credential provided requires login verification.

To better review the application, please provide working credentials or OTP for us to accurately follow your test instructions and replicate them.

While we’ve made some progress, we are still far from achieving our goal. The main issue with the review process nowadays is providing credentials for testing. Previously, we could create test users on the Facebook Developer platform. However, this functionality has been “temporarily” disabled for several months. There is no ETA when we can create test users again.

Our first attempt involved creating a fake Facebook account. The challenge was that we needed special permission to support users with Meta Business Suite, requiring us to connect the fake user to it. This action immediately triggered the suspension of the fake account and a validation request to prove the account was genuine. Validating a fake account is not feasible:/

The only reliable way to proceed with the review is to provide credentials for a personal Facebook account, which seems unreasonable. To facilitate the reviewer’s access, we disabled all security measures, such as two-factor authentication (2FA) and login notifications. Despite these efforts, Facebook’s security mechanisms blocked the reviewer. The reviewer tried to log in from the Philippines while we were in Poland.

We re-enabled 2FA, generated recovery codes, and resubmitted our review on the same day with additional information.

June 27

Our new App has been rejected once more. Despite the reviewer’s attempt to log in, Facebook’s security mechanisms blocked the user, even when he had access to the OTP recovery codes. We received a notification on our mobile devices indicating that Facebook blocked a suspicious login attempt. On one hand, we commend the robustness of Facebook’s security measures. On the other hand, we have a challenge allowing the reviewer to log in and test our App.

Because we started the App review, we discovered that we have access to Direct Support, where we can ask Meta representatives questions directly. This option was previously unavailable to us. We posed two questions:

1. Why is our App restricted, and when will Meta remove the restriction?
2. How can we avoid blocking the reviewer from accessing the Facebook account?

We had high expectations for this direct support channel and were pleased with the prompt responses we received. Unfortunately, the support representative informed us that:

• They are sorry about our situation and understand our inability to proceed.
• This support channel is only for the App review questions. We should report the bug. However, bug reporting is currently limited to Meta Business Partners.

It could have been more helpful. At least we learned why we can’t submit bugs. We checked the Meta Business Partner program only to discover that:

None of your Business Manager accounts are eligible for Meta Business Partners.

Unfortunately, our App does not meet their allowed Partner criteria. There is another dead end here.

Before receiving the support response, we decided to resubmit our App for review in its current state, hoping the reviewer might have better luck on their next attempt.

Shortly after, we received feedback for our second question to Direct Support:

We noticed that you have already submitted your application, and it is currently pending review. Therefore, we kindly request that you wait until the review team completes the process.

They reiterated that if the issue persists, we should submit a bug report, a process unavailable to us as non-Meta Business Partners.

June 28

It was a final breakthrough day. In the afternoon, we received several emails about new violations found in our restricted App. We checked the App Dashboard on the Developers’ Platform and noticed it is no longer restricted! Finally!

We didn’t receive any specific emails regarding the lifting of the restriction. The new violations pertained to some missing information in our Privacy Policy. Since the information was already present, the reviewer must have missed it. We quickly updated the Privacy Policy to make it more readable, resubmitted the instructions, and within an hour, we received a message confirming everything was in order.

The remaining violations were related to the “Consumer Connection” section. The violation was accurate because we had replaced the App with a new one. Our initial plan was to use the old (now unrestricted) LightWidget App to query the widgets in the background and gradually migrate users to the new LightWidget for consumers App as the tokens expired over time. All newly connected Instagram accounts will use the new AppApp. Unfortunately, we can’t have both apps running simultaneously under the same website.

We decided to keep the new App and remove the “Consumer Connection” functionality from the old App. Having two Apps instead of one will benefit us in the long run. If one of the apps gets restricted or encounters issues, we will still have the second App operational. Relying on a single app could bring our entire service down, as it did this time. The downside is that all consumer connection accounts must migrate to the new App. We have already started sending expired token notifications to facilitate the migration. We apologize for any inconvenience this may cause.

We also added all the widgets to the refresh queue so we can finally send requests to the APIs. The script fully completed the global refresh on June 29 after midnight. The restriction issue is now finally resolved.

On the same day, we received an alert in the Facebook app about a login attempt from the Philippines. Considering the time-zone difference, it was nighttime there (22:00 Philippine time), making it unusual for someone to be reviewing the App. Nevertheless, we decided to allow the login.

Soon after, we received an email confirming a successful App review.

So now we have three apps:

1. LightWidget – The unrestricted version.
2. LightWidget for consumers – The replacement for the consumer connection.
3. LightWidget for business – A potential replacement for the business connection.

We decided not to enable the replacement for the business connection, preferring to keep the old, unrestricted version while retaining the replacement as a backup.

Lessons learned

Unfortunately, we can’t control issues that might occur on Meta’s end in the future. We hope these will be rare, but they are beyond our control.

We have learned the importance of clear communication with our users during such incidents. While some suggested emailing license holders, this approach has its challenges. Developers often purchase widgets for clients and may no longer be in contact with them. Given that we have orders spanning over ten years, it can be challenging to track the current owners of the widgets. Additionally, due to GDPR data anonymization, we don’t have all the original email addresses. We’re also unsure if we have consent to send such emails, especially to those who no longer use our services. Sending emails to such users could lead to a potential violation of the GDPR.

To address this, we have decided to enhance our email notification system. In addition to expired token notifications, we will now include a newsletter-style update about ongoing issues with Meta, required migrations due to changes in the Instagram API, and major new features. We hope these notifications will be infrequent, particularly for downtimes. You can opt-in for these updates on the Settings page.

We have also improved our website by including a global notification bar for announcing issues and changes. During the incident, we displayed downtime notifications only on the Dashboard and a few other pages. We created these alerts hastily and with little planning. We were definitely not prepared for such an event. Next time, the alert will be more visible. Below, you can find an example. We will pin important messages to the top of all pages.

Additionally, we have developed new tools for our support desk to help automate responses and address emails more quickly.

Refunds

We understand the impact and frustration caused by our services’ downtime, especially for those who purchased upgrade add-ons just before the issue began.

According to our Terms & Conditions, we have a 30-day refund policy. Due to the downtime, if you purchased an add-on within 30 days before the issue started (from May 7, 2024), you can request a refund until August 7, 2024, extending the policy to 90 days. Please submit a support ticket with your order ID for a refund.

End word

We want to thank all users who showed compassion and understanding, offering much-needed kind words and support through comments, messages, and support tickets. We appreciate your patience and support.

As always, we strive to provide the best service possible. We sincerely thank you if you choose to continue using our services despite this incident.

Sincerely,
The LightWidget Team