FieldFX Root Cause Analysis

Beginning on September 25 at 10:54 AM Central, customers began receiving errors when trying to load the report viewer in Salesforce. Mobile reports were unaffected.

Service was restored at 12:04 PM.

We deliver many of our services through a Content Delivery Network (CDN). We originally wrote and hosted our own CDN to meet our unique needs. As our business scaled and the commercial CDN market matured we decided to switch to a world-class CDN solution using Amazon AWS (Amazon Web Services) services.

The performance and uptime of the new CDN is a significant improvement over our older solution. We have been transferring our services to the new CDN throughout the last year.

On September 25, we deployed an updated version of the report viewer to meet some needs expressed by our customers. The deployment consists of changing the setting on the CDN to point to the updated version number.

We made a mistake when migrating the report viewer service to the new CDN. One request was being made to the new CDN and another request was being made to the old CDN resulting in mismatching parts of the report viewer.

Our resolution to the problem was delayed because shortly after deploying the updated version of the report viewer, we implemented some necessary housekeeping in the AWS account. Part of that housekeeping involved deleting unused permissions.

Soon after being notified of report viewer problems, we devoted all our efforts to restoring the deleted permissions. Only after we restored all the backups and noticed that the problems persisted did we investigate further, eventually identifying the actual cause.

The most obvious solution is to complete the migration to the new CDN. In addition, we are implementing a BETA channel for the report viewer. All future report viewer deployments will go through the BETA channel first, and customers will be notified of the BETA deployment. Customers are advised to test report functionality against the BETA channel and report any issues discovered. That way, any issues can be addressed before deployment to the normal Production channel.

Beginning on May 23, 2023, at 4:45 PM Central, customers began receiving errors when trying to use the Report Viewer. Service was restored at 5:35 PM.

We deployed the latest version of the code, which included a check for the ReportsAsFiles feature. The feature was added to our managed package approximately 18 months (about 1 and a half years) ago.

The information is stored on a custom metadata table. Depending on how you install the package, permissions for the metadata table are either automatically assigned to all users or need to be assigned individually.

An incorrect assumption was made that package installations were typically performed using the “all users” option, which would have automatically granted the necessary permissions to all users. The query to check for ReportsAsFiles did not include a check for permission errors and as a result, the issue was never caught during any of the testing phases of the code.

To test managed packages, we will start installing for “admins only.” Additionally, we will individually assign permissions to new objects/fields to users.