Developer mode requirement on Windows

[LysandreJik]

The current snapshot_download and hf_hub_download methods currently use symlinks for efficient storage management. However, symlinks are not properly supported on Windows where administrator privileges or Developer Mode needs to be enabled in order to be used.

We chose to take this approach so that it mirrors the linux/osx behavior.

Opening an issue here to track issues encountered by users in the ecosystem:

• i was trying to create custom tokenizer for some language and got this as error or warning.. transformers#19048

[BramVanroy]

Thanks for making this issue. Responding to Lysandre's comment here huggingface/transformers#19048 (comment):

No, to use WSL you do not need to enable developer mode since 2017. I have been running WSL2 with CUDA and never had Developer Mode activated. I also only ever use WSL2 if I really cannot get software to run natively on Windows. WSL is great but it has its pitfalls/quirks. Like the interaction with PyCharm of setting up/managing different environments, but also IO which is really slow between /mnt/ and the distro's native drive. I prefer native Windows when I can.

So for most of my transformers projects, I've used Windows, mostly without issue or I posted a PR to fix the problem. Windows users seem plenty. And in my case, I am especially worried for the class room. At the start, I use Colab because it's easy for beginners. But after a while we switch to a local installation and running scripts instead of notebooks so they at least know about those things. I cannot require them to enable developer mode (liability) and WSL will take too much time of a class to get working for everyone.

Outside of the classroom, you can think about anyone who are just getting started with transformers and want to play around with it on their own system - gamers, hobbyists, enthusiasts, or anyone with an interest in AI-related things. Windows users are still the majority (even among developers) and they are not all hardcore PC enthusiast. Even if they are (like me), they may not be eager to have this additional step to allow access to transformers. So my issue here is that instead of making access to AI easier (as Hugging Face has done in the past, for which I am of course grateful), this new change actually makes it harder for a portion of users. I am not sure if the advantages outweigh the disadvantages because everything worked fine before.

[henk717]

For our frontend and many others as well this will be a big issue, it is very hard to explain to users as to why they need developer mode or administrative permissions in order to be able to download a model. This seems very suspicious to a user since they will not be aware of the technical limitation. It would be much nicer if we have control over this, such as being able to use the old cache system if desired.

[ebolam]

Posting here to keep an eye on this. Fully agree with the sentiment. Requiring developer mode will limit the ability to use transformers in programs that are given to non-developers. An average user will not want to enable developer mode when it's full of warnings.

[julien-c]

I'm curious if this affects many many Windows users, or not. Let's see if more users post about this in the coming days (to see how we could fix it)? Will also ask on Twitter=)

[BramVanroy]

I'm curious if this affects many many Windows users, or not. Let's see if more users post about this in the coming days (to see how we could fix it)? Will also ask on Twitter=)

As @henk717, @ebolam and my posts suggests, the problem is not just with developers. If they really need to and you do not give them a choice, I guess they will enable developer mode or be forced to switch to WSL (but again: I was not eager to enable dev mode on my machine either; it comes with its own risks).

A major problem is the consequence for non-developers: hobbyists, students, end-users who try to run products/demos/tools that use transformers under the hood. Those won't know what to do or will not want to start enabling things in Windows that give you a big security warning. Those are exactly the new people you'd want to get interested, not scare away. The results of both your Twitter poll and the comments on Github will therefore be very biased towards developers.

As @henk717 suggests, keeping the option to use the old cache mechanism would be great.

[julien-c]

The results of both your Twitter poll and the comments on Github will therefore be very biased towards developers.

Yes you're right @BramVanroy

[mrseeker]

Adding my comment here (as suggested by @julien-c). Activating developer mode is asking for trouble. On top of that, I had to drop support for WSL because my developer PC simply cannot handle it (running it will take 75% of available RAM + impossible to shut down once activated, not to mention that I cannot even run dropbox on my windows machine without constant hanging). Turning on WSL and developer mode not only requires users to run a beefy pc, but it also pushes them to security warnings and dialogues that I Pavlov'ed normal users into to pressing "no" for security reasons.

[nickmuchi87]

Following this closely as I faced the same issue I posted in the HF forum here:

https://discuss.huggingface.co/t/symlink-error-when-importing-ortseqclass-model-via-pipeline/20706

I am not a developer and was not too comfortable with running PC in dev mode, tried experimenting to see if that would solve it but got the same error after switching dev mode on and doing a full restart.

[henk717]

A major problem is the consequence for non-developers: hobbyists, students, end-users who try to run products/demos/tools that use transformers under the hood. Those won't know what to do or will not want to start enabling things in Windows that give you a big security warning. Those are exactly the new people you'd want to get interested, not scare away. The results of both your Twitter poll and the comments on Github will therefore be very biased towards developers.

Very much this, transformers is a dependency of multiple user friendly or end user experiences. KoboldAI in our case, but on steam there are games like AI Roguelite and AIDventure that also use transformers as a dependency. Explaining a user who buys an end user product in a store front that they manually need to change Windows settings for the program to work correctly is a step to far and should not be automated away since you are then changing security settings.

So a solution that does not require system wide changes is very much desired for applications that are either portable, or installed trough automated means.

[mrseeker]

Another thing: It's windows now, but what if you would force the user to use root to install the package under Linux? Would you force people to use "sudo" to run transformers, knowing that it uses pytorch which has an exploit in their system that you are currently scanning for?

[LysandreJik]

Hey, thanks all for your comments. We agree that it shouldn't behave this way under Windows and we're working on a solution.

[Wauplin]

Hi everyone and thank you for the feedback.

Here is a PR to implement a cache-system variant that do not need symlinks if there are not available: #1067. See description for more details. Any feedback is very welcomed ! :)

[Wauplin]

#1067 is now merged and will be available in next release of huggingface_hub. For now one can pip install from the main branch to run on Windows.
I'll keep this thread updated once the release is made.