2020-11

Authenticating with Azure Datbricks API as a Service Principal

Hello! Back when I first started working with Azure Databricks I used bearertokens to authenticate both on my dev machine and in Azure DevOps pipelines. It wasn’t especially challenging to add a masked variable containing a bearertoken that never expired to a pipeline and then pass that in. However over the past 12 months or so things have matured to the point where we really should use premium workspaces, because on the standard tier workspaces everyone is an admin and that’s the only permissions level.

2020-11-05

2020-06

Export External Tables from Azure Databricks Workspace

Hello! I recently needed to export the “CREATE” statements for any hive tables on an Azure Databricks Instance whose paths were set externally to an Azure Datalake. Other htan changing the output_folder and hte name(s) of your mount points below this script should run on any Workspace. There is a little work to be done with regards to the order of objects saved to the csv files, as views require the tables to be created first before they can run, but seeing as I only had to run this once I manually re-ran the small number of views that failed.

2020-06-30

2019-12

Unexpected Modifications to Notebooks Azure Databricks

Hello! I’ve been working a lot of Azure Databricks, and one of the things I’m really impressed by is how straightforward it has been to use source control to store not just notebooks, but other artifacts such as jobs and clusters. This is because the API is well documented, and it’s a case of pushing either a file (in the case of notebooks) or a bit of json config (in the case of everything else).

2019-12-02

2019-11

Set Azure DataBricks BearerToken to Never Expire

Hello! Found another trick with setting the expiry date to access tokens in Azure Databricks to never expire, and I don’t recall seeing it in the documentation but it is there as a tool-tip, and who ever reads those! In Azure Databricks these Personal Access Tokens allow users to access the Rest API; however a token must first be created manually after a Workspace has been created. Note that tokens can be created and revoked via the API, however the fact is that a token must first be created manually prior to using the API.

2019-11-06