Data Analytics – Abylon

Data warehouse project using the Abylon Rapid Platform

Zoltán Zicherman — Wed, 04 Oct 2023 10:32:50 +0000

Automated SQL Data Warehouse project done in 3 weeks with minimal prior experience

Data warehouse implementation - easier than expected

As a junior BI consultant with 10 months of work experience, I was given the task to build a complete data warehouse, data marketplace and populate it with data.

Considering that I had little experience in the field, this task promised to be a time and energy consuming challenge. I did have some help from a senior colleague who monitored and supported me during the process, but this did not change the fact that I had to understand and complete a complex process without any prior experience.

All I had to do was:

understand the steps of the process,
define the structure of the data warehouse, the objects to be fetched from the source system,
and put them into a simple Excel-based configuration file.

By setting up the configuration file properly, the creation of the new data warehouse could be almost completely automated.

The parameters I needed to set up the configuration were the availability of the different source systems, the objects to be fetched from them, any changes to the objects (e.g. columns of a table, data type of a field, name), how they are loaded, how they are historized and other optional parameters, the steps of the ETL processes that load the data, and the availability of the target system. Having all this in one place allowed me to quickly check and modify the structure if changes were needed.

After definig the metadata, the Rapid Platform was able to automatically create the different database layers (stage, data warehouse, data marketplace), by running a PowerShell command, on the specified target server, which in my case was an MS SQL database server.

The benefit of the solution lies not only in the ease of use for Business Intelligence developers, but also in the significant time saving (and therefore cost reduction) that it brings. The time and energy freed up through the automations could be spent on deeper analysis of the task and the data, and through that on tasks that actually create business value, such as designing and refining the right data model and subsequently producing analytical reports on those models.

A few words about the environment

In a Microsoft stack-based development environment, a data warehouse automation solution can be easily implemented with the Abylon Rapid Platform.

Quick overview of process of generating a data warehouse and the software used during the project:

Using MS Visual Studio, the configuration files for the data structure, the server connections and the project environments, can be easily accessed and modified. There is also a configuration file that stores the connections and environments.

After loading a specific tool from the Rapid platform, I used a simple PowerShell command to generate the files defining the STAGE, DWH layers as part of the solution.

After creating the build in Visual Studio, running another Rapid Platform command, the given data model structure is deployed to the given SQL server (if the connections are defined correctly). I then created the different OLAP models, which I could build in Visual Studio as well as in Tabular editor. I could then deploy this to an Analysis Services server using an separate command.

After that, I only needed to transfer the data. The “how” I already defined in the configuration file. The stored procedures for transferring data from the STAGE layer to the DWH layer were automatically generated. The invocation of these stored procedures can be simply added as “execution plan items” to the configuration file as part of various “Execution Plans” that can be easily invoked and scheduled by a SQL job. The manually created stored procedures that fill the DM layer can also be added to the part defining the ETL process inside the configuration file and then can be run as part of the “Execution Plan” when runing the SQL job. This allows the loading of the entire data warehouse with a single job execution.

This created an automatically updating data warehouse with the necessary data historization available, as well as the OLAP cubes. After connecting to the latter I could starts generating the various analytical reports.

Despite the fact that this was the first time I was using the Rapid Platform, and generally speaking I did not have much DWH experience, thanks to the flexibility and simplicity of the platform, the whole process, including the subsequent improvements required by the client, did not take more than 2-3 weeks.

The main benefits I have experienced

In addition to taking a huge burden off the shoulders of the developers, data warehouse automation is also a significant benefit for the customer, as it can greatly reduce the time and effort spent on an DWH implementation.

The benefits of automatisations

Ease of use

Flexibility

SSAS Tabular model management with the Abylon Rapid Platform

Zoltán Zicherman — Mon, 23 May 2022 15:25:22 +0000

Intro

In the context of a number of recent enhancements, we thought it is worthwhile to write up a brief summary of the Tabular model services available within the Abylon Rapid Platform and highlight how they support development, processing and logging.

Development

First of all, during development, tabular models can be deployed to any number of environments just like ETL processes, databases, reports (etc.). There is no need to rewrite the connections every time you move to a live environment, the platforms will do it for you after setting up the appropriate parameters. It also provides the possibility to publish user-level self-service models to a productive environment.

The developer’s life is made much easier by automating the creation of dynamic measure tables. Dynamic measure is useful for any business, but the creation and maintenance of the necessary tables and measures is time-consuming and sensitive to changes. Abylon Rapid Analytics takes this burden off the developer’s shoulders. It also makes it easy to produce measures based on selected measures such as LY, MTD, ALL, etc. based on calculation groups.

Processing

Our execution plan manager allows the usage of several OLAP processing steps in each execution plan item. Such as:

Full process, this is obvious
Dynamic partitioning, code-driven version: dynamic partitioning creates the partitions according to the configuration and processes the appropriate number. Main features: rolling window, incremental partitioning, incremental processing, fix interval.
Dynamic partitioning, data mart-driven: the data market developer has the possibility to create his own stored procedure, which passes the processing and creation command to the framework.

Logging

Metadata backup: main data of SSAS tabular models installed on environments, up to the query level of partitions.

SSAS instance monitoring: load, queries monitoring with Platform Planning functionality.

This is just a short teaser because this list could go on and on. We believe that, as with the whole Rapid Platform, in this case our strengths lies in the convenience of the ready-made solution, the cost-effective customization and the fast, customer-oriented development.

The available features do not set a mandatory direction, but provide opportunities, and the developments made during implementation help to fully meet the needs of our customers and provide support and good foundation for further BI development projects.

The post SSAS Tabular model management with the Abylon Rapid Platform appeared first on Abylon.

Case Study: Advanced Analytics on top of a legacy on-prem DWH

Zoltán Zicherman — Tue, 08 Feb 2022 10:50:09 +0000

Introduction

In this post we’re going to describe an advanced analytics project we recently delivered for a large international company. The main goal of the project was to improve the management of our client’s retail data (sales data coming from their shops) by adding an additional advanced analytics platform on top of the existing infrastructure.

Business Challenge

Prior to this project all retail data was only managed by a large, traditional, monolith on-prem SQL data warehouse. Data coming from this legacy data warehouse is aggregated and presented in pre-made group-level dashboards which are limited and highly dependent on report developers. Accessing more detailed data or data outside of these dashboards, or creating custom reports based on it would require a lot of effort.

This rigid infrastructure was not suitable for advanced analytics which requires dynamic data and an option for self-service business intelligence (BI). Furthermore, our client has other BI solutions that would also require this type of dynamic data.

Solution – Advanced Analytics Platform with Azure Synapse and Databricks

We built a new, cloud-based (Azure Synapse-based) data warehouse on top of the existing on-premise data warehouse and connected it to the existing Databricks (DBX) Platform. We extract the raw data from the on-prem data warehouse on a daily basis and make it accessible in our new Advanced Analytics Platform.

The new platform consists of two separate sub-platforms:

A Databricks (DBX) Platform for development and Business Value Realization solutions (for self-service development)
A Synapse Platform for transactional level analysis (for self-service analysis)

The Databricks Platform is a self-service development platform that uses Python, Machine Learning and Artificial Intelligence technology. It’s mainly meant for internal developers for forecasting and creating various data science solutions, and for external vendors (developers) who deliver custom solutions to cover their own business needs.

While the Synapse platform’s main goal is to enable self-service data analysis. Users can directly connect to the raw data from the data lake, execute custom queries, see the data on a transactional level and even connect to the data from Excel or Power BI.

More dynamic data

Compared to the aggregated dashboards (built on the on-prem data warehouse), in the new Synapse-based data warehouse all the data is available on transactional level. This means data like: cart size (how many items are bought together), all transactions, transactional data on item level (which items are bought together, discounts, sales quantity, etc) all meta-and master data (shop details, like opening hours, plot size, geo coordinates etc.) Most of this data was previously NOT available in the pre-made dashboards built on the legacy on-prem system, and now it is available in the Synapse environment, in a more dynamic set-up and to a much larger audience.

Easier access, larger audience

With this new advanced analytics platform, business analysts can also access all the data and create their own reports, even from Excel. This setup also opens the door to create an additional data layer for third party companies (like vendors who are selling their -typically beverage- products in these shops), which can even be sold as a service, to create an additional stream of revenue.

Scalability and reduced infrastructure costs

Another huge advantage is that the new platform allows our client to dynamically scale compute resources up or down depending on their needs. This means a significantly better solution in terms of cost effectiveness.

The post Case Study: Advanced Analytics on top of a legacy on-prem DWH appeared first on Abylon.

Advanced Analytics Platform – how to create a dream framework for Python/R

Zoltán Zicherman — Sun, 30 Jan 2022 17:16:24 +0000

Introduction

Advanced Data Analytics and Data Science is a really hot topic right now and it’s something definitely for the future as well. In 2020 the global data science platform market was valued at USD 31.05 billion, and it is expected to reach USD 230.80 billion by 2026, registering a CAGR of 39.7 % during the forecast period, 2021-2026. Impressive numbers, but how does it affect your life? Data is getting more and more important in nowadays world, and even you can play an important role in its expansion.

Since you clicked on this post it’s clear that you are interested in data analytics. Maybe you’re interested in the technical side (as a tech expert), or in the results of data analysis (as a manager) or you are just simply curios about this topic.

You would like to start working with some data, but your company doesn’t have the data infrastructure and the workflow for that yet. You’re not the only one who’s in this situation. As data science is going viral, more and more companies are realizing its benefits and thinking of investing in it on long term, but they don’t exactly know how to create a complete framework around cool buzzwords like “Python, ML and AI”.

In this article we describe how to create one. Since we have a lot of experience in working with industry-leading Microsoft products, in this example we are going to show you a setup that is based on the Microsoft cloud architecture, which is already tested, implemented and operated on a daily basis.

At Abylon Consulting we’re crazy about data, so if you like this topic browse through our blog for some other data analytics related posts and guides, or check out our product or service offerings if you are looking for a solution for your company.

What are my essential goals?

I want to work with Python/R/spark, since I know they are interesting and very fashionable nowadays. I mastered basic SUM, COUNT, AVG Excel functions long ago, I want something way more sophisticated.
I’m interested in more massive datasets: even public dataframes from the internet, even internal database tables or corporate files.
I want to visualize my results on good-looking reports, spiced with some automations.
I want to have a complete infrastructure that is robust and simple at the same time

To meet these expectations, we should get more familiar with the componets that deliver these features.

Introducing the components of our Advanced Analytics Platform

Good news, that all these needs may be solved with a few resources of Microsoft Azure:

Power BI (PBI)

Possibly you are already familiar with Microsoft’s industry-leading data visualization tool.
If you like charts, filters, plots, you will feel home. Call it a well-balanced mixture of Excel, Power Point and SQL.
Though it’s not placed in MS Azure, you will need it to create beautiful and informative reports.

Azure Databricks (DBX)

A Jupyter notebook-like interface which does the core data analysis and processing.
You can write scripts in Python, R, Scala, and even SQL. Supports Apache spark also.
It runs in the cloud, you don’t need to install any distibutions/libraries/etc on your computer.
Scalable compute resource, scalable cost (flexible and predictable, pay as you go)

Azure Datalake Storage (ADLS)

The cloud storage you will use. Simple, but very useful, consider it as a more strict OneDrive or Google Drive.

Azure Data Factory (ADF) – optional

If you need scheduled data copy from external/internal source systems, ADF will possibly be handy.
It is responsible for data movement activities in the cloud, or even on-prem.

High-level design

At the figure above you can see these components of the infrastructure. This basic design can help you to imagine the role and their consecutive manner. Let’s get through it in a nutshell:

First, you need to have some source data stored on the ADLS to begin with the analysis. Don’t worry, you can upload any flat files (like csv, json, xlsx) from your computer, there is no need for ADF at the moment.
You will consume & analyse this data with DBX, and store the output on the same ADLS.
If your analysis is done, and the result data is available on the ADLS, it’s time to open up a PBI file, and bind the necessary data from the source file(s) of the ADLS.
If you are finished with report development, you can publish your report to PBI Online, and share it with anyone.
If you have build a solution that is time-dependent, and needed to be refreshed from time-to-time, you can automate the execution of the DBX and the refresh of the PBI report in the cloud.
- If not only the calculations are time-dependent, but source data should also be refreshed, it’s time to add an ADF. With ADF, you can schedule data loads from source to sink (ADLS), plus DBX notebook executions can be triggered as well

Implementation

Azure tenant

To get all of that, you need to have az Azure tenant and an Azure subscription. You can try it for free for 12 months and you can use a credit of 200 USD in the first 30 days.

Resource Group

After the registration, create a new Resource Group (RG) which functions as a box to store your things. It doesn’t cost any money, it is only a simple “folder” in your subscription.

DBX

Create a new Azure Databricks (DBX) workspace, within this RG. Lauch the workspace, and take a look around there. You can find the menu options on the left, and maybe the two most important options are Workspace and Compute.

Workspace stands for a folder structure for your executable Jupyter notebooks (ipynb format), while
Compute holds all the compute resources (clusters), which execute your code in the notebooks

You will have to create a cluster prior to any development. You are charged by cluster time usage, so you are supposed to optimize the runtime and not leave any idle time. You can pick whichever setup fits you and your budget best.

Check this deeper guide with a case study if want to dive in DBX Python coding. Remember, core coding will be done here in DBX, all other components only help you with connectivity, data freshness, and robust operation. If you wish to try DBX alone at first, you can do it without creating any other resources. But, for a complete platform, you will need other components as well.

ADLS

Create a new Storage account (ADLS), also within the same RG. At Settings/Endpoint blade (left-side menu options are called “blade” in Azure) you can find the links to this specific ADLS, use this link when you aim to connect to this from PBI.

Connect ADLS with DBX

To make these two resources communicate with each other, it is needed to create a bridge between them. The best solution is an OAuth 2.0-based solution, named Service Principal (SP). This SP is an Azure Application, that creates the bridge between the two components, providing authentication from and to both parties.

The recommended option is a mount point, which is basically a link to a container of an ADLS, that the DBX notebooks can use as a link to point directly to the files placed on the ADLS.

Having a working SP provides a rather dynamic communication between the two parties allowing you to quickly access (write/read/modify) any files stored on the ADLS from DBX code. This allows you to create a flexible Machine Learning or Big Data solution, with serious inputs and outputs.

Power BI

From Power BI it is very easy to connect to ADLS, pick the ADLS Gen2 connector, sign in or provide your access key (preferably in a corporate network) available on Security+Networking /Access Keys blade of the ADLS.

Publish the report if you are finished, and set scheduled refresh in the online PBI service if it makes sense (based on your data).

ADF

If your source data changes or refreshes from time to time, you may need an ADF to solve the data processing. Create an ADF in the same RG you used before, set it up and get comfortable with it in a few steps.

You can create pipelines with activities, most likely a copy activity, of which you can set ADLS as sink, and your preferred data source as source. Both cloud and on-prem sources usually have authentication that you need to comply with the following way.

Create a Linked Service to store the connection information about the source system. It will basically stand for an independent data store. Every high-level information like id, connection string, username and password, other credentials are usually used here.
Create a Dataset within a Linked Service to specify the exact details of the source files/table you want to connect to (both source or sink). Typically foldername, filename, text encoding, schema name, tablename, other low-level details are used here.

Since ADF is in the cloud, for on-prem data loads you need to configure an Integration Runtime (IR) on the physical machine you are using. Either you are using recurring flat files from your home PC or refreshing database tables from the company’s on-prem SQL server, you will need to have a self-hosted IR installed on the physical machine to comply with the connection details.

Real-life example

If you are curious to see a real life implementation of a similarly built Advanced Analytics platform and the benefits it offers compared to traditional data warehousing check out this short case study: Advanced Analytics platform on top of a legacy, on-premise Data warehouse.

Some more tips & tricks, know-how and best practice!

Well done! As the architecture is complete, everything is ready to operate in production. All of these components have some more useful features to know about, briefly:

ADLS

- Security: Access Key cans be changed, reminders might be set
- Lifecycle mnagement: Data storage tiers may be set (hot-cool-arhive) that determine data access time and the mostly cost

DBX

- Cluster version: runtime setting is rather easy e.g. from version 10.0 to 11.0, just open the dropdown menu
- Install libraries: easy to install any Python/R/etc by PiPy, Maven, CRAN or from external link
- Access libraries on the cluster: Notebooks can only use the lib, if they are runing on the cluster on that the lib has been installed
- Data source: you can configure other source than ADLS only

ADLS-DBX SP

- Connections can expire: check at the SP (Azure App) when is it expiring, usually 1 year, but it can be set longer. To renew it, the whole process has to be done again (except Azure App creation).

PBI

- Data sources: it is smarter to use validation by access key, and use only one data source instead of many, if new files are from the same source, this case won’t mean extra work
- PBI service: to schedule a refresh, access keys must be added here also (again, smarter to use one source so that credentials are not to be added multiple times)

ADF

- Scheduled triggers: set periodical execution in the lowest detail one can imagine. Run pipeline on the 2nd and 3rd thursday of every odd month? No problem to set, very cool.
- Alerts: very useful to set alerts on pipeline run failures, it helps to get known about problems at the right time
- Debug vs Triggered run: triggered run can execute processes in paralel and has greater compute capacity than simple debug. As the name indcates, use debug only for debugging
- Git: use Azure DevOps for code management. Releases, publishes, merge, pull, can be done very well.
- Azure KeyVault: store your secrets in this Azure resource. Mostly ADF focus, but can be handy many other cases

Still curious for more?

If you feel that your appetite is just coming for more advanced analytics solutions, check:

Azure Synapse Analytics – if you want to quench your hunger with modern cloud database solutions and massive data workloads in the cloud, or take a bite to
Azure Machine Learning – if you feel like taking an adventure with the next generation data science solution of Microsoft (an extraordinary crossover of DBX, Rapid Miner and a REST API portal)

The post Advanced Analytics Platform – how to create a dream framework for Python/R appeared first on Abylon.

What clients say about Rapid Analytics?

Zoltán Zicherman — Tue, 16 Nov 2021 17:29:58 +0000

Interview: What our clients say about our DWH Automation tool

A few weeks ago we successfully completed a data warehouse project for Dashboard Solutions using our Data Warehouse Automation Solution – Abylon Rapid Analytics. In relation to this, we had a quick interview about with senior BI Architect Zoltán Nyircsák to get a quick feedback from the clients’ side.

Abylon: Thank you for taking the time for this quick interview. First of all, for what tasks do you use Abylon Rapid Analytics?

Zoltán Nyircsák: It is important to mention that we basically did not buy a technology or a component, but a solution. We chose Abylon and their data warehousing solution because the technology they used promised a quick rollout (and they kept that promise). But nonetheless, it was paramount that the experts were truly experts from both business and technology perspective. In fact, on the customer side we don’t even really need to understand what ARA (editor’s note: ARA stands for Abylon Rapid Analytics) does, it’s enough to articulate the business problem and then enjoy the benefits.

Abylon: How and to what extent does Abylon Rapid Analytics simplify your work?

Zoltán Nyircsák: Since the technological solution and its maintenance and development are not on the customer’s side, so it’s not easy to answer this question. However, we do enjoy the benefits from the business side. Eg: rapid development of data markets.

Abylon: How responsive is the solution (or Abylon?) How fast can it track changes?

Zoltán Nyircsák: The responsiveness is good. There were several source system changes that were easily handled by the solution. For example, there have been several overnight transitions to develop a new source system. Because we work with source systems that are in use, they are constantly changing, evolving, getting new features. So it was important for us that the BI solution we were building could handle this (with the help of Abylon experts, of course.)

Abylon: How flexible is the system?

Zoltán Nyircsák: So far, we have found a solution for everything together. Here again, I would like to point out that the technology used is “hidden” from us. This division of responsibilities between us and your experts was agreed prior to the project. Our part is to formulate the request, whether it’s some specific data, analysis, or business problem, and you give us the solution.

Abylon: How complicated do you find managing such a system?

Zoltán Nyircsák: This is also part of the service, so we don’t have to deal with it. However, the data warehouse and data market based on the solution, and the Power BI service, was an excellent choice. We are able to respond quickly to new business needs, facilitating the accurate, timely and reliable delivery of not only data but also information.

Abylon: What do you think, how much time did the use of Rapid Analytics saved you during your data warehouse building project?

Zoltán Nyircsák: Without it, it would’ve been an order of magnitude (I really mean the 10 times multiplier) more time and money to develop such a solution and then to adopt and react to all the changes. The development time has been largely reallocated and instead of spending time on manually creating the obligatory things, we were able to spend more time on the essential things and put more energy into the construction of the data market.

Abylon: Finally, what would be the feature that would make Abylon Rapid Analytics even more valuable to you?

Zoltán Nyircsák: Right now, we don’t really need anything else, but if I think about the future, maybe a built-in report on the operation of Abylon Rapid Analytics, the data loads and performance would be good.

If you are interested in our DWH Automation solution, head to the dedicated product page or contact us and we’ll be happy to answer your questions or arrange an online presentation

The post What clients say about Rapid Analytics? appeared first on Abylon.

How to post G/L records with retroactive data with the help of BI and SCD?

Zoltán Zicherman — Wed, 10 Nov 2021 15:04:57 +0000

For whom this blogpost is meant for

IT professionals who work with data, optionally familiar with BI or Datawarehouse technologies, and may need to create a solution in which they need to create posting files or an interface for accounting /ERP systems from historical data. This article can help them to identify the required actions and things to consider.

On the other hand, it is also intended for financial professionals or Business Analysts who are involved in a project or projects where the goal is to develop an interface between normal transaction systems and accounting/ERP systems.

Intro

What is posting in accounting?

If you know what an SCD, or even a SCD2 is, feel free to jump on to the task description. If you don’t know what they mean, the following few lines can help you understand.

As part of the month-end closing process, it is always necessary to make numerous manual postings in the financial accounting. Posting in accounting is when the balances in sub-ledgers and the general journal are shifted into the general ledger. Posting only transfers the total balance in a sub-ledger into the general ledger, not the individual transactions in the sub-ledger. In other words, after you enter, review, and approve transactions, you must post them to update your systems with current transaction records and maintain system integrity.

What is an SCD?

SCD stands for Slowly Changing Dimension, and is essentially a data structure that allows us to keep track of changes in our data.

What does “data change” mean?

In the world of IT, we typically store data in relational and non-relational databases. In this article, we start with relational databases, which are typically stored in SQL databases. Data entry is usually done through some kind of application where users input data using a desktop or sometimes some mobile device. No matter how, but the data eventually ends up in a data table. Consider a typical transaction: a sale event occurs and the following data (also called a record in a database language), that is, records are created

A data changes occur when a change, modification or deletion occurs in the already created and stored data (in this case the data above). SCD types are intended to symbolize the handling of the above changes.

More info on SCD:
https://www.datawarehouse4u.info/SCD-Slowly-Changing-Dimensions.html

https://www.allthingssql.com/slowly-changing-dimension-types/

Slowly changing dimension – Wikipedia

For example, SCD0 means that no changes are allowed to the previous data – (in practice, this is how transactions are stored in Block chain technologies, where you cannot delete or modify previous transaction data).

SCD1 is when we pass all changes to the current data file and do not store anywhere what data was stored before. Technically, SCD1 can already be considered a historization.

In this post, we show one of the benefits of SCD2 and how this type of historization can be used to create appropriate monthly posting files or a proper interface for accounting. You can read more about SCD2 and the different SCD formats following the links above

The task: Create a posting file with retroactive – aka. closed periods adjusting

Let’s suppose our company builds a calculation engine (hereinafter: the engine) in a database that receives transaction data from an application and then recalculates the transaction values from the received transaction data each time it runs.

( It doesn’t matter if this is a separate database, a database schema, or part of a transaction system to be developed, this solution is feasible in any case. I will now describe a solution where the transaction data is transferred using ETL processes to another database, and some of the calculations take place there, not in the application itself )

Several transformations can occur in the database that do not happen on the application side, e.g. Exchange rate assignment, assignment of partner data, product info, even purchase data, purchase prices, etc. On every run the engine receives the data from the application and from all other source systems and completely re-calculates the output data.

In our case, it is also the job of the engine to produce a posting file (or an interface) for the accounting program at an unspecified time at the beginning of each month. This part of the process is done manually, there is no automated running and data extraction

The Challenge

The engine completely recalculates everything on each run – clears the entire table contents at the database level (TRUNCATE) and loads the calculated data completely (INSERT).

Posting, on the other hand, occurs once a month at an unspecified time, during which only a portion of the calculated data (typically data for the previous month) is loaded into the accounting software (or ERP), based on a predefined field containing the transaction date.

The business analysis revealed that some transaction data may change even months after the first recording (UPDATE).

We did not receive any information regarding the possibility of data deletion during the development, however, it cannot be completely ruled out from the engine’s point of view

So to sum up the problem: how can we ensure the consistency of the calculated current data and the accounting data at the analytical level in the long run?

Let’s see our Example:

The transaction system will be launched in January 2022, and will be showing the following transactions when they are imported on the 2nd of February:

On the 3rd or March 2022 (the time of the next monthly posting/import) the whole transaction data set will look like this:

During the import on 3rd of March 2022, according to the protocol, only the data with the February TransactionDate (aka. PostingDate) will be loaded into the accounting system, however, we can see in the example that the data line (record) with the ID 3 has been deleted and for the record with Id 2, the Quantity was changed from 50 to 48.

The Alternatives

During the posting (in the posting file), we can follow the general cancellation rules, ie we make a change by adding the value of the previously recorded transaction with a negative sign (we can also post a canceled transaction in this way), and then re-enter the value and data of the correct transaction.

If the accounting software or ERP allows it, it is advisable to record the fact of the cancellation, and to record the ID linking the posted item to the original transaction together with the accounting item. (In our case there are two options: it is logical to use an ID field for this, however, if for some reason this field is not available, the InvoiceNumber and ItemNo lines can clearly identify a record, so we could use those as well).

A possible solution would be if the accounting system would always receive the full calculation, but this would cause a lot of data redundancy that would only make data reconciliation between the accounting and the engine even more difficult.

So keep in mind to make it as easy as possible to reconcile the differences.

The Solution: Historization and SCD2 helps!

First of all, we introduce historization into the engine in at least SCD2 format. During historization, it is sufficient to historize the final table containing each calculation step.

Historization – illustrated with data below – is done by introducing 2 Flag fields as well as two date fields, a DAT_FROM and a DAT_TO field, which show the validity period of the given transaction record.

The two flags are FLG_IS_CURRENT and FLG_IS_DELETED. Flags are managed as Bits, so they can only take a value of 0 or 1, 0 is false and 1 is true. (This form of SCD2 is just an option, no obligation for the data types, nor for the column names, etc. Try to grasp only it’s logic, if you are not familiar with it.

For the historization you will need to select the key field or key fields, using which the items can be clearly identified. These key fields determine whether an item was previously included in the data or not, and the modification steps are also performed based on these key fields.

When entering the system the current data will get the current date / time in the DAT_FROM field, the value of FLG_IS_CURRENT will be 1, the value of FLG_IS_DELETED = 0, and the value of the DAT_TO field will either be empty, ie NULL or e.g. 2999-12-31.

(I suggest a preferably high default value for the DAT_TO field, because NULL’s are mostly incomparable in rdbms systems.)

If an item is deleted, FLG_IS_CURRENT will be 0, FLG_IS_DELETED will be 1, and the DAT_TO value field will be updated to the time of the change.

If modified, the value of the FLG_IS_CURRENT field for the modified record changes to 0, and the value of DAT_TO changes to the time the change is registered (when the modified field is historized), and the field with new values is entered as the current new value in the historized table.

It is a good idea to store the current and historical tables in separate tables.

In the above case the HISTORY table would look like this on the 3rd of March 2022:

We can see that both items from January, which were modified, were modified in February, after the date of the last dispatch/import (February 2)

Preparing for posting/import

In order to be able to correct the posting with previously deleted or modified items, whether we export them manually or automatically for posting, we must record the date of the export. Let’s call this the DateOfExport parameter.

It is advisable to organize the actual solution by supplementing the record files containing the current transactions together with the plus sets needed for the corrections (and providing the sets with the appropriate signs), which can be handled to perform all the modification and transformation steps in the engine together which helps to produce the posting file.

The following figure will help you understand which sets to add to the current one for this purpose:

Explanation:

The blue and red areas are the ones that output the entire dataset that needs to be transformed.
The blue areas are the ones that are counted with their original sign, the red ones are calculated with the opposite sign.
It can be seen that only part of the entire ACTUAL file, is exported. The export is based on the original PostingDate field of the transaction data, that is, when the transaction was created.
The @LastDateOfExport is the date when previous month’s data was exported.
In addition to the current monthly data of the current record file, we need to add three sets of data:
- Deleted records: Records for which the following conditions are true in the historized table (each condition must be met, i.e. there is an “and” relationship between the conditions):
  - FLG_IS_CURRENT = 0
  - FLG_IS_DELETED = 1
  - DAT_TO › @LastDateOfExport – this guarantees that items that have been deleted AFTER a previous export will now be entered with a negative sign in the record set before the transformation.
- Historical data for modified records with a negative (opposite) sign: these are the records for which the following conditions are true in the historized table (each condition must be met, so there is an “and” relationship between the conditions):
  - FLG_IS_CURRENT = 0
  - FLG_IS_DELETED = 0
  - the value of @LastDateOfExport is between the DAT_FROM and DAT_TO records
- Current data of modified records: all records from the entire current set whose identifiers are included in the records defined in set 2. These are not duplications because they only contain records that were already current in a previous post, but their DAT_TO is greater than their @LastDateOfExport. The sign of the values of the present set is not changed.
One important modification: for all non-current records, the PostingDate value, ie the date value based on which it is selected for the given monthly post export, is changed to DAT_TO, thus guaranteeing that the added sets are included in the actual post file

The base record set generated this way:

We handle the above sets as a single input and we perform the transformations that are required for the sets to be accepted by the accounting software or ERP.

This blog entry does not cover this topic because these transformation are all target system specific and do not affect the data set and historization operations described above.

Benefits of this approach vs. other approaches

We don’t do a complete data file load into the accounting software or ERP system every time we do a posting. We do not burden the accounting systems with unnecessary records, simplifying the reconciliation.
We ensure continuous matching at the analytical level between the accounting software / ERP system and the reports showing the current data (eg towards partners)
We facilitates analytical reconciliations — especially when IDs are included in the posting file so that cancellation events (items recorded with the opposite sign) can be marked separately. This also facilitates coordination with partners and monthly and annual closing tasks.
We can also use this in case of automated connection. The only task is to store the DateOfExport value (recordeed manually in the present case study) in the “engine” after the successful posting.
We cover almost every possible business need creating a robust solution that is prepared for future requests. For example, even if initially there has been no demand to manage cancellations, if such requirement may arise later the engine will be prepared for this

Just one last thing:

Our Rapid solutions (Rapid Planning and Rapid Analytics) provide the ability to use all of the slowly change data management methodologies at the data warehouse level, so our company and our clients can easily create data market solutions supplemented with historization, based on which “time-machine” tabular models can be created. For example our datawarehouse automation solution (Rapid Analytics) is ideal for default historicization solutions

Thank you for your time! I hope you liked my post and found it useful. If you have any comments, additions or questions feel free to contact me at kreiszzs@abylon.hu

The post How to post G/L records with retroactive data with the help of BI and SCD? appeared first on Abylon.

Railway data map visualizations

Zoltán Zicherman — Mon, 11 Oct 2021 15:29:12 +0000

In the world of Power BI, there are no identical solutions. Both, the business requirements and the data are almost always different. In the ETL processes as well as in data modeling, we need to find the solutions that can provide the answers to the questions in the simplest and fastest way.

The complexity of the data processing, the running time, the transparency of the tabular model, the complexity of the measures used and the running time are also very important metrics measuring the quality of our work.

When visualizing railway data on a map, the appropriate model can be identified knowing the business meaning of the data and its volume and complexity. Typically, there are two types of data available in case of line-based infrastructures:

Point
Route (Railway route

Connecting the fact data with the map objects

Point data can be an extraordinary event (eg accident), data related to an infrastructure element (maximum speed, condition, scheduled maintenance, cost) or detailed traffic data for a certain train based on its railml schedule and actual traffic data. This data can be fitted to a star schema, and the dimension table serving the map can be linked to the model with a simple One to Many connection

In the image below, the EU_OBJECTS dimension table is the display source of the map, it will display the objects (line or point) broken down by ID, based on their type. The EU_TRANSPORT table contains the factual data:

Note: Abylon MAP requires the usage of a unique ID and and object type, so it can be mapped to geojson files, or individual lines can be displayed separately by direction. Click on the button below for more details.

Organizing routing or interval type data into a model

Traffic data that contains a measured value along its entire route is called reference data. Tihis includes passenger traffic, ticket sales data, travel from A to B, or train data that do not change from the start point to the end (eg weight, railway company), but do not include traffic time data or values calculated from them.

We have several choices regarding this data. The problem is similar to period-type data, and there are three available options:

Interval data is broken down into the smallest point-like data required:

Like from time interval to days, and then from relations to line segments and stations. The days of the time interval can be extracted into separate rows in the fact table and these (as points) can be added into the data model.

Tabular On the fly:

We entrust the data model to sort out the data using one or two connection tables. See later.

DAX on-the fly:

When working with small amounts of data, instead of complicated modeling or significant data multiplication, we can filter using a DAX filter, in which case the dimension and fact tables are not linked within the model. Continuing with the time interval example: calculate ([something],filter(’Fact’,[datfrom]<=max(’date’[date]) && [datto]>=min(’date’[date]))

Combining solutions 1 and 2 to find a calculation optimum.

In this case the fact data rows are not broken down to an elementary point / section level, but to a higher level (e.g. railway line) and the model itself remains in accordance with number 2. (author’s note: in my opinion, this is the best method, we can provide detailed, pin-level data with it, but more frequent queries tailored to business needs can occur faster).

We will now discuss solution number 2 in more detail.

Tables underlined in yellow color provide the dimensional expansion:

The row of the fact table contains data for complete routes, to which is attached the routes M2M table, which contains the unique values of the route identifiers.

Routes are connected with a many-to-many relationships to the MAP object level, whose final dimension table contains the map data: coordinates, names, and so on.

The Fact data is like this:

ID	FROM	TO	DIRECTION_ID
123	A	D	23
124	K	O	24

The directions M2M connection:

DIRECTION_ID

The MAP technical table is for extracting the directions:

DIRECTION_ID	MAP_OBJECT_ID
123	A
123	B
123	C
123	D
124
124
124

This approach allows you to assign data to an elementary map object without multiplying rows.

So, if we filter for an individual object in the MAP’s dimension (e.g an elementary section of line), the engine will calculate all of directions that have this element in the route description, and the fact data will be filtered for them

Which approach is the best?

In the case of point data, the question seems clear, although we have already seen a case where a set of factual data was related to a railway station and also to the related section. With railway data models, it is particularly interesting that you need to handle the relation to a railway line itself, so the master data of the map should contain the stations that belong to several lines multiple times.

In terms of where to start with relative/related data, the quantity and quality of the data is the decisive factor. With that knowledge (and clear business requirements) an experienced data modeler can easily chose the right approach.

If we set aside the railroad example and look for a more straightforward example, successful management of the time interval may still be the right example

Business Case	Solution
Short time intervals (1-3 days)	It is recommended to transform the time interval to a point regardless of the amount of data, as it does not cause an increase in the size of the rows of factual data.
Long time intervals with limited data (such as contract records)	In this case, the model-based on-the-fly and the daily breakdown would result in unnecessary duplication of rows, but the the dax on-the-fly approach is a good choice and we can treat the factual data with interval-filtered measures based on an independent date table.
Medium / variable time intervals, large amount of data (hotel reservations)	If our calculations show that breaking down the data into days would increase the number of rows by an order of magnitude (minimum 5-10 day average time interval), it is recommend to use the model-level on-the-fly approach.
Medium / variable time intervals, with extremely large amount of data or special requirements	In some cases, it is necessary to try to keep the size of the tables implementing M2M connections manageable. In this case it may be worth combining the different solutions: the lines are broken down into weeks we create explanatory tables for the weeks and for the combinations within a week. In this case, there will be 28 rows for a week, which can provide a favorable breakdown even for a multi-year interval.

Custom Power BI map visualizations for railway companies

Abylon MAP is built on top of Power BI offering, custom, advanced map visualization features for companies dealing with line-based infrastructures. It is ideal for railway (or other transportation companies dealing with line-based infrastructures) looking for an affordable, customizable, easy-to-use map visualization tool for Power BI.

Infrastructure-specific Power BI map visualizations
Visualization of line-based networks using both circles and lines
Full integration with Power BI (built on top of it)
Active maps based on Power BI models or GeoJSO

The post Railway data map visualizations appeared first on Abylon.

Multi-level access rights in Dynamic Row Level Security

Zoltán Zicherman — Fri, 13 Aug 2021 13:22:05 +0000

In today’s blog post we will discuss in detail the implementation of hierarchical dynamic RLS with Power BI without many-to-many connections.

Case Description

Our client’s request was to implement a dynamic RLS. We had proper table specifying username and authorization pairings/access rights. Based on the business needs a hierarchical management had to be implemented, something like this:

Country
- City
- Employee

In the fact tables each row includes a worker ID and a city ID. It is geographically assured that a certain city belongs to a certain country within the defined period. Based on the client’s request the rules for the visibility of fact and dimensional data are:

Senior managers can see the data of one or more country’s data (and their cities) and the data of the workers who work there
Area managers and sales manegers can see the data of one or more cities and the workers who work there
Employees can see their own data
There are administrators who can see everything
There may be employees who, apart from seeing their own data, can also see data generated in a city (eg sales and area manager at the same time).

This is the summary what we see in the logic that serves the RLS:

The problem

The solution described on the Microsoft site is pretty and nice, but does not solve the issues highlighted and underlined in the rules described above (in the previous paragraph).

Source: https://docs.microsoft.com/hu-hu/power-bi/connect-data/desktop-tutorial-row-level-security-onprem-ssas-tabular

The occasionally necessary many-to-many connections are easily doable using intermediate tables, but this approach is far from ideal in terms of modeling and performance.
A special problem are the users who “appear” in two categories/on two levels (the last two rows in the table above), because in their case you need to filter two dimensional tables, but due to the AND relationship of the filters, the user permission are not extended, but narrowed down, so instead of having a combination of the two data sets you only get a section of it.

The extended model (seen below) shows that an additional problem is the visibility of related dimension data, for example, the cities or countries dimension does not filter employee data, i.e. it is not guaranteed that a manager with city-level authority will only see the master data for employees who worked in his city

It is guaranteed that you will be lost among the various many-to-many switchboards and even if you do it, still:

• It is complicated
• therefore, it is difficult to test
• any corrections or modification carry a huge risk

The solution

Principles

Leaving behind our habits and past modeling practices, we have laid down new principles that provide a solution to the business need without over-complicating the model or the whole process:

We used a single dimension table to filter based on fact RLS conditions: Country, City, Employee
We used a single table containing user data to define the RLS: UserId, Access level etc.
Given that the number of returned value sets is low (filter for less than 10 country / city / employee IDs), we allowed ourselves to use DAX filters instead of m2m relationships in the model.

Preparation

Fact and dimension

You need to generate an unique identifier in the fact and dimension tables that ensures the transfer of RLS filters, which ensure that a city can belong to a country, so that the employee ID and the city will be connected:

concat([employee_id],’-’,[City_ID])

The source of the dimension table can be the fact table queried using distinct, or in the case of a corresponding HR strain, an extract of it. It is important that an identifier appears only once in the new dimension table and that the table must contain the data to be used in the report, eg name, telephone number, city coordinates, etc.

In case of more detailed demand it can be supplemented with validity management, and a period identifier can be appended to the identifier.

Viewed in PowerQuery, our dimension table will look like this:

The RLS table will be the summary table shown at the beginning:

Based on this the new model will be much more simple and clear

The new model contains:

all RLS data
all employee data
every city
data for all countries
the factual data to be examined, with a simple relationship

DAX Role Filtering

Basis

We assume that the access levels have separate AD groups, so the Role containing 3 filters can be added separately (at the end we will examine what if they don’t).

Logic

Filtering for Country:

The dimension table must be filtered for the rows that contain the country IDs that appear next to the user ID in the RLS table:

Conversely:

Collecting the country ID-s related to the users from the RLS

var usercountries=calculatetable(values(’RLS’[COUNTRY_ID]), ’RLS’[AD]=username()))

Dimension filtering for these ID-s:

filter(’Dim Employee-City’, ’Dim Employee-City’[COUNTRY_ID] in usercountries)

Implementation

Rules added to the Dim Employee-City table.

Country:

’Dim Employee-City’[COUNTRY_ID] in calculatetable(values(’RLS’[COUNTRY_ID]), ’RLS’[AD]=username()))

City:

’Dim Employee-City’[CITY_ID] in calculatetable(values(’RLS’[CITY_ID]), ’RLS’[AD]=username()))

Employee:

’Dim Employee-City’[EMPLOYEE_ID] in calculatetable(values(’RLS’[EMPLOYEE_ID]), ’RLS’[AD]=username()))

If there are no separate AD groups

The three rules can be combined into one.

’Dim Employee-City’[COUNTRY_ID]

in calculatetable(values(’RLS’[COUNTRY_ID]), ’RLS’[AD]=username()))

’Dim Employee-City’[CITY_ID]

in calculatetable(values(’RLS’[CITY_ID]), ’RLS’[AD]=username()))

’Dim Employee-City’[EMPLOYEE_ID]

in calculatetable(values(’RLS’[EMPLOYEE_ID]), ’RLS’[AD]=username()))

Summary (Lessons learned)

Although it is basic rule that we solve problems in a model or data source preferring SQL, aiming for the simplicity in case of DAX formulas, here from the model’s usability point of view the complex solution is less resource intensive:

working with two tables instead of using a complex model
we ensure low result numbers during RLS filtering (eg it does not return the ID of all employees in the country, but only 1-2 country codes or 1-2 cities)
the model remains simple, stable
We were able to create a general solution for the customer and subsequently this RLS solution has become a standard for several other data sets with similar / identical dimensional resolution.

The lesson of the solution is that many complex problems can be solved within the Microsoft BI ecosystem, no matter what tool we are using (SQL, .NET, Tabular or DAX), we just don’t need be afraid to step out of our schemas, standards, and try to think “outside the box” to find the ideal solutions.

The post Multi-level access rights in Dynamic Row Level Security appeared first on Abylon.

When it’s NOT the right time to build a data warehouse

Zoltán Zicherman — Mon, 19 Jul 2021 17:31:51 +0000

I was recently asked by one of our clients whether they should start building a data warehouse at one of the companies they acquired or not, and if yes, what are the arguments in favor of building it. This conversation inspired me to write this article.

(Note: I used to work as an economics and business professional before switching to IT and Business Intelligence. In this article I try to build a bridge between economics and IT, so apologies to all the IT professionals if, for the sake of non-IT readers, I simplify or even over-simplify some of the things mentioned here.)

When is NOT the right time to start building a data warehouse?

If you just started creating reports, building a reporting system

If you are just starting to put together your first real reports, it is very likely that even you do not know yet from which data and exactly how you should come up with the numbers you need to present to management. In this case, it is not worthwhile to start dealing with data warehousing or Business Intelligence yet. Most likely, in the process of reporting and analyzing the data, you will already get to a point, when:

You will unify and simplify, for example by developing a common data dictionary
You will do some OLTP-based development, for example unify the data collection (eg by implementing or expanding an ERP system) and move it to a common platform
The content of the reports may not be accurate yet, so the purpose of the reports themselves will change constantly.

If you have already started creating reports, but for now a few (let’s say 3-4 complex or 5-6 simple) reports is enough to cover all the tasks required by your company.

All the topics seems to be covered by a few reports and they are flexible enough to manage possible changes, and lastly, changes in the basic data do not need to be immediately synchronized with other systems.

If the size of the data is not very large (and at least one of the previous two scenarios are true)

If the number, size and complexity of the data used in the reports (eg the number of records or rows in a table) is not so large that a more serious optimization (or reporting system upgrade) would be necessary in the period between two reports.

If the frequency of data requests and the size of data related to them is NOT overloading the source systems (eg an ERP system) during the reporting

In terms of technology, BI systems have been brought to life by the fact that systems primarily designed for data entry/input (so-called OLTP systems, such as ERP systems, which have been optimized to make data entry easier for users) have not been optimized to serve a large number of queries. If you do not overload the source systems, i.e there is no significant slowdown during the data entry or data query process, then there is no urgent reason yet to start thinking about data warehouses or BI systems.

When SHOULD YOU start thinking about building data warehouses?

If you are drowning in the sea of Excel and / or Power BI reports

If you create a lot of reports, that vary in complexity, utilized technology (e.g. PowerQuery, Excel transformations, Power BI data modeling), and it is becoming harder and harder to effectively debug them, furthermore, it is hard to synchronize the reports with changes to the source system precisely because of their complexity, you should consider moving toward a standardized solution.

If you need a historical aspect to your corporate data

Sometimes, an OLTP system (e.g., an ERP system) is not suited to put data in a historical perspective, e.g. you can’t simply check when an item number was in stock and when it wasn’t or check which team a former employee was part of or when did he leave the company.

If you need a stable, fast connection for data of different quality, source and depth

Some reports require you to compile data consisting of millions of records (millions of rows) from different source systems (e.g., an SQL database and an Excel, or mass data available through a REST API, etc.). In such cases, especially if process speed is requirement, it is unavoidable to use a more powerful SQL database customized specifically for reporting needs.

The size of the data, the time allowed for report creation, or the frequency of the reports requires it

You process large amounts of data? Or you need reporting with very short time intervals (eg hourly report update)? In both cases you may start to think about investing in a data warehouse.

You need to run queries frequently and / or they are so big in size that they put a heavy load on the source system engine (e.g. a database engine)

In such cases, it is logical to start looking for a suitable data warehouse solution. Also, if the frequency of the reports is a pain point, you might check out our “Near Real-time data warehouse” blog article, that describes in detail how Abylon Rapid Analytics clears this hurdle.

The post When it’s NOT the right time to build a data warehouse appeared first on Abylon.

Data Warehousing in 2021 – old problems, new solution

Zoltán Zicherman — Mon, 21 Jun 2021 15:45:13 +0000

With the spread of new, innovative data solutions with demand mainly in the domain of business intelligence (BI), data science and general automation, company leaders and stakeholders are encountering an old problem, data warehousing (DWH).

The majority of companies jumping onto the bandwagon still use legacy data warehousing solutions. Moreover, they usually have data quality issues and do not have the data engineering and DevOps capacity to fully utilize the power of data.

We all know the old data saying “garbage in, garbage out” or “GIGO” for short, and the solution is always the same: clean the incoming data as much as possible. Of course data standardization can be time consuming and at the end of the project there’s no time left for solving the real business problems. We also must admit that these are not the most exiting things to do, but unfortunately you really can’t avoid these steps if you want a properly working data warehouse and data solution built on top of it. Or could you? Well, the good news is, that you actually can

Data Warehouses are built from multiple data sources and complex Extract, Transform and Load process (ETL) processes. Maintenance and development are both costly and time consuming. Furthermore, both require deep knowledge of the running process and data sources. This technical knowledge equals to expensive hard-to-train (and replace) specialists who spend most of their time with unfulfilling mundane tasks.

But we have some good news!

Abylon Rapid Analytics is a data warehousing framework built on Microsoft Azure Cloud products and services making data warehousing more efficient, faster and more flexible. Rapid Analytics also supports Cloud, Hybrid and On-Premises setups making it easier to fit your business needs.

Painful and boring “re-inventing the wheel” steps are replaced with a metadata-based data warehousing, allowing the developer team to focus on core development, creating value from information while requiring less support from the DWH team, allowing them to focus on their domain.

Below is a comparison of a traditional approach (let’s call it “manual” approach) with a project done with Abylon Rapid Analytics

Task	Traditional Approach	With Abylon Rapid	Increase in efficiency
ETL Design	20 man day	5 man day	75 %
ETL Development	100 man day	5 man day	95 %
Stage Development and optimization	10 man day	2 man day	80 %
DWH Development with versioning and optimization	50 man day	10 man day	80 %
Automated testing and debugging	50 man day	10 man day	80 %
DevOps	30 man day	3 man day	90 %
Monitoring	20 man day	2 man day	90 %
Total	280 man day	37 man day	85 %

If you feel that you are making the same steps again and again, having boring hours with low efficiency, then you should familiarize yourself with modern Cloud solutions such as Abylon Rapid Analytics.

Also, check out this blog post to see our solution in “action” during a real project: https://www.abylon.hu/blog/real-time-data-warehousing/

The post Data Warehousing in 2021 – old problems, new solution appeared first on Abylon.

Data Analytics – Abylon

Data warehouse project using the Abylon Rapid Platform

Automated SQL Data Warehouse project done in 3 weeks with minimal prior experience

Data warehouse implementation - easier than expected

A few words about the environment

The main benefits I have experienced

The benefits of automatisations

Ease of use

Flexibility

More on the Abylon Rapid Platform

Author of the post

Follow our social accounts for more news and technical insights!

Found this post interesting?

Other popular posts

Five Biggest Challenges in Financial Planning

Master data custom validation using the Rapid Platform

Unleashing the Power of Excel: From VBA to Power Query

UK market here we go – Abylon’s International Ascent

SSAS Tabular model management with the Abylon Rapid Platform

Intro

Development

Processing

Logging

Case Study: Advanced Analytics on top of a legacy on-prem DWH

Introduction

Business Challenge

Solution – Advanced Analytics Platform with Azure Synapse and Databricks

More dynamic data

Easier access, larger audience

Scalability and reduced infrastructure costs

Advanced Analytics Platform – how to create a dream framework for Python/R

Introduction

What are my essential goals?

Introducing the components of our Advanced Analytics Platform

Power BI (PBI)

Azure Databricks (DBX)

Azure Datalake Storage (ADLS)

Azure Data Factory (ADF) – optional

High-level design

Implementation

Azure tenant

Resource Group

DBX

ADLS

Connect ADLS with DBX

Power BI

ADF

Real-life example

Some more tips & tricks, know-how and best practice!

Still curious for more?

Author of the post

What clients say about Rapid Analytics?

Interview: What our clients say about our DWH Automation tool

How to post G/L records with retroactive data with the help of BI and SCD?

For whom this blogpost is meant for

Intro

What is posting in accounting?

What is an SCD?

What does “data change” mean?

The task: Create a posting file with retroactive – aka. closed periods adjusting

The Challenge

The Alternatives

The Solution: Historization and SCD2 helps!

Preparing for posting/import

Explanation:

Benefits of this approach vs. other approaches

Just one last thing:

Author of the post:

Railway data map visualizations

Connecting the fact data with the map objects

Organizing routing or interval type data into a model

Which approach is the best?

Custom Power BI map visualizations for railway companies

Multi-level access rights in Dynamic Row Level Security

Case Description

The problem

The solution

Principles

Preparation

DAX Role Filtering