Automatic Smart Sync takes the hassle out of syncing, protects user data, and gives users peace of mind.
Overview, Settings Options, Auto Sync Triggers, Manual Sync Triggers, Order in which Deltas Sync, Legend of Sync Icons on Web, Data Conflict Resolution, Mobile Survey Sort Order on Survey List Page, When a User Encounters Problems, Error Handling, Known Issues
Sync between mobile devices and server has been completely refactored using a “granular” approach that syncs discrete units of data called “deltas”.
The previous survey sync approach required the sync to complete as a whole after many consecutive data sync requests.
- If one request failed, the entire sync failed.
- Data was susceptible to corruption or ending in an incomplete state.
- Wildnote was unavailable for other use while a sync was in progress. Large amounts of data could require a long period of stable connectivity.
- Logging and debugging errors were impeded, large requests could timeout, and users who waited too long to sync increased risk of data loss.
To overcome these things, the granular approach means:
- Every edit is tracked and synced one discrete piece at a time, on a per-observation/answer basis.
- When the device has connectivity, after the user taps “Save and Exit”, the deltas immediately and automatically begin syncing to the server.
- Mobile users can watch the countdown of the deltas as each survey syncs.
- Wildnote is available for full use while the sync is in progress. Users can even edit the survey while it is being synced. The app detects additional changes and adds them to the stack of deltas to be synced.
- Web users see icons which indicate when all mobile survey observations have been received, photos are loading, or a survey integrity check has failed.
- Separate Project Sync and Sync All operations are gone because the auto-sync processes new projects, project updates, survey forms, taxon lists, project locations, etc. (See User Support section for detailed order.)
- A table of deltas is stored on the mobile device as well as the server, which enables us to retrace data history on both mobile and web so we can better address data integrity issues.
All settings options can be found under the 3-dot menu on the project listing page of the mobile app.
- Auto-Sync Sync On/Off: Disable the automatic sync with this button (i.e., a user with a limited cellular plan may want to wait for wi-fi access to allow syncing or the user knows that they have extremely poor connectivity).
- Show Survey Time by Default: This switch toggles display of the time survey was created and the Survey ID number.
- Sync All and Reset: Synchronizes all data with the server. NOTE: This is only to be used under the guidance and direction of Wildnote Support.
- Tapping Save & Exit button on any survey.
- [Android only] On app startup. iOS does not sync on app startup.
- Each time Save & Exit is tapped, the app will attempt once to connect to the internet, then abort if there is no connection. It will try one time per Save & Exit.
- Only one operation is performed at a time during sync (no concurrent/multithreaded activity).
- Inside a survey: When user taps “Save and Exit”, the focus returns to the survey list page, a red number of deltas appears over brown arrows for that survey, and the deltas begin decrementing as they sync to the server.
- Since photos sync last and are slower to sync than the other data observations, these final deltas will take longer to complete than the previous ones. Users can eyeball how many photos are left to sync as they count down.
- NOTE: user can abort a sync by exiting the app.
Manual Sync Triggers
- Outside a survey: Pull down to refresh: from the Survey list page or Project list page, scroll to the top of the list. Place finger at the top of the list of surveys or projects and drag finger down across the list and release. The sync will begin. A spinning icon appears until the sync completes.
- OR tap on the brown arrows at the top of the Projects page.
- User can also re-initiate the sync by opening another survey and tapping Save and Exit.
Order in which Deltas Sync
- Projects (includes locations, member roles/updates, taxon favorites)
- Survey Statuses
- Survey Templates
- Checkmark appears when observations are complete.
- Photos are processed last
- While photos are processing, the icon on the web survey list page is a photo; when complete, the icon returns to checkmark.
- Scheduling changes manifest as surveys being downloaded
- Project<->Survey Template mapping table is generated
- Project Documents that may have been modified during sync
- Changes to taxonomy lists, download indicator statuses
IMPORTANT NOTE: The green checkmark now appears on mobile to confirm that all deltas for that survey were successfully communicated to the server, but does not convey when their processing on the server successfully completes.
Also, after the checkmark appears, photos may still be uploading from your mobile device to the cloud. And the sync to mobile from the server may continue for photos, taxon list, re-sort of Wetland species repeaters, survey ID, and other info. The sync spinner remains animated until these complete. Users are encouraged to QA/QC their data as soon as they are able.
Legend of Sync Icons on Web Survey List Page
Sync Status Symbol
Sync is complete for all observations. (Sometimes has not yet accounted for photos.)
After install/migration of the sync refactor release, checkmark appears for all data successfully synced with sync refactor (created/edited), regardless of platform.
While sync is in progress:
(For more info, see section above, Order in Which Deltas Sync)
Meanings for no checkmark:
-See [empty icon], below.
-For roles other than superuser or project admin, the survey will only show when the checkmark is ready to appear.
Upload of photos to cloud storage is in process
Failed integrity check-the survey may have missing questions.
Flow of icons during survey sync:
1.Non-photo data syncs first; triangle icon displays because questions remain to be synced.
3.Next photos sync, during which the photo icon displays.
4.After photo sync completes, checkmark reappears.
For more info, see section above, Order in Which Deltas Sync.
Sync interrupted due to connectivity problem.
Check connection and reinitiate sync to resume.
This icon will persist, even when sync resumes. It remains until the sync completes.
A survey in this state is probably also failing the survey integrity check because some data has yet to sync.
Meanings for no checkmark:
Data Conflict Resolution:
The former sync paradigm handled changes at the survey level, so only one set of changes to a survey could be preserved. Automatic Smart Sync handles data changes at the object level, which allows any number of users to successfully sync changes to different questions (or to the same question for photos and multiselects) in the same survey.
If two users make changes to the same question, the syncs will succeed, the deltas will be preserved, but the last change made will win.
Example of conflict resolution:
If User1 makes a change earlier than User2 but syncs later than User2, User1’s delta will not be applied. (The time the data was actually changed matters, as opposed to the time it was synced.)
IMPORTANT NOTE: Should the incorrect data be overwritten, please contact Customer Success (firstname.lastname@example.org) and ask for the data to be restored. Please include the name of the project and a link to the survey along with the details of which question needs to be fixed.
- Survey date descending
- Location name ascending
- Survey form name ascending
- ID descending (as a proxy for created at)
- Blank location sorts to the top
NOTES: While being synced, surveys have no user and will sort randomly in the set of today's surveys for the current user. They will not necessarily be at the top of the list.Sort by date does not include the timestamp, so recently synced surveys can be randomly sorted among all surveys created by the user on that day.
When a User Encounters Problems
- With sync refactor, when a user encounters problems, syncing may often be the best way to solve it. This is a paradigm shift from previous Wildnote versions, where avoiding sync was wise to prevent data loss.
- Users may be confused when the sync spinner continues after actions visible to the user have completed; this is no cause for concern. Other kinds of data proceed to sync after the green checkmark shows up (it confirms all non-photo answers have been synced to the server). Following the green checkmark, photos, taxon list and other info may still be syncing.
- If the sync stalls and pull-down won’t restart it, exit the app, reopen and pull-down to initiate sync.
- If the sync process as a whole fails, or an individual sync operation fails due to lack of connectivity, the entire process will be aborted. The next scheduled or triggered sync will pick up the process where it left off.
- If a delta fails to sync due to an error response code from the server, it will be marked as errored and pushed back to the queue.
- If a delta fails to upload three times in the one sync process, it will be removed from the upload queue and treated as an errored delta in future syncs
- If the app enters the background while a sync is in progress, the sync will be immediately aborted, and the user will need to manually restart the sync.
- Survey ID may be delayed in showing up on the survey list page. It sometimes does not appear until after the sync completes and the green checkmark appears, especially on a large survey. It does eventually appear. To hasten it, refresh the page (pull down or tap brown arrows on title bar, or sync another survey). Symptom: survey may receive green checkmark, but survey ID remains 0
- Chained conditional logic ability to hide/show chained conditional logic after sync is broken. It's ok if you don't change the trigger.
- Erratic deltas may occur on some LG devices
- Video and Document answer types have been disabled.
- Counter: When a web-created survey with a Counter question has an empty value and is synced to Android, the Counter’s +/- buttons don’t work until you enter a value.
- Location: For a web-created survey with a Location question that was synced to Android with an empty value, when it is edited on the web and synced, it won’t show on the Android survey list but the answer does appear in the survey.
- As of 5/19/22, some surveys with repeaters may show the wrong structure on mobile for a period of time, but straighten out some time later after sync data processing completes.
- iOS: Intermittently, the UI on the survey list page may cut off the form name/location vertically. This happens after syncing some surveys created on iOS with long names. Such surveys may also flash during sync due to a wrapping issue with the survey name. Tapping the screen usually ends the flashing. The best fix is for the user to shorten the name of the survey form.
- Survey sync limit: When the survey sync limit is decreased (i.e., from 50 to 20), the app does not delete the extra surveys from the device. Having more data is not a bad thing, and the surveys still reside on the device.