Ayhem@lemmy.world to Artificial Intelligence@lemmy.worldEnglish · 1 year agoWebsites blocking OpenAIlemmy.worldimagemessage-square13fedilinkarrow-up163arrow-down14file-text
arrow-up159arrow-down1imageWebsites blocking OpenAIlemmy.worldAyhem@lemmy.world to Artificial Intelligence@lemmy.worldEnglish · 1 year agomessage-square13fedilinkfile-text
minus-squaresenoro@lemmy.mllinkfedilinkEnglisharrow-up6·1 year agoCompanies do honour robots.txt. Maybe not the small project made by some guy somewhere. But large companies do.
Companies do honour robots.txt. Maybe not the small project made by some guy somewhere. But large companies do.