I have been trying to motivate my team towards using Ray with our snowflake setup. I recently came across this anyscale blog – "Introducing the Anyscale Snowflake Connector" which excited my further. After some searching on github, I could find the code for this connector as well.
I am a complete complete newbie to ray and hence couldn’t get this connector to work consistently. However, I am thoroughly familiar with snowflake and its python connector and I have a complete understanding how a snowflake connector in ray should work, so I went ahead and wrote my connector while seeking motivation from the above codebase as well as from the ray custom datasource documentation.
:drum: GitHub - prabodh1194/ray_snowflake :drum:
Please go through my repo and fire away with questions and PRs and ideas on how can we go about using these.
Thanks a ton.
P.S.: I’ll be spending sometime this week to create an optimal write step which can directly write an arrow table to snowflake instead of taking the Pandas route.