The feature My Work is based on Microsoft Delve and provides the users with the most relevant content based on who they work with and what they're working on. The information in My Work is tailored to each user.
IMPORTANT: My Work doesn't change permissions and users will only see what they already have access to.
For further information regarding data security please check this Microsoft article.
The more you and your colleagues use Microsoft O365 to work together, by viewing, editing and sharing each other's documents, the more useful My Work (Delve) will be for all of you.
Currently you cannot disable My Work completely, but you can switch off the 'By my colleagues' part. If you wish to disable 'By my colleagues', you can control access to Delve from the SharePoint admin center. By default, users in your organization have access to Delve.