Problem with loading multiple CSV files that add up to 4m rows of data

El Cid

Member
Joined
Aug 22, 2016
Messages
52
Reaction score
0
Points
6
Location
Greenville, SC
Excel Version(s)
Excel 2016
So I've used the built in tool in PQ to do this and it's a really slow process I save it as a connection. All of the CSV files have to exact same format. I ran across the M Code listed below on another forum where the respondent claims you can just save it to the advance editor and it'll run. It seemed to fit my need, so I loaded it and it stopped with a table to list error where csvFiles...I changed that to Table.SelectRows. I'm not sure how to fix this, even if that's possible. Anyone have a clue? Thanks

let
// Set the folder path
folderPath = Excel.CurrentWorkbook(){[Name="PATH"]}[Content]{0}[Column1],

// Get list of CSV files in the folder
files = Folder.Files(folderPath),
csvFiles = Table.SelectRows(files, each [Extension] = ".csv"),

// Create a function to load CSV files
loadCSV = (filePath) =>
let
csv = Csv.Document(File.Contents(filePath),[Delimiter=",", Encoding=65001, QuoteStyle=QuoteStyle.None]),
headerRow = Table.PromoteHeaders(csv, [PromoteAllScalars=true]),
sourceName = Text.BeforeDelimiter(Text.AfterDelimiter(filePath, "\"), ".csv")
in
Table.AddColumn(headerRow, "Source", each sourceName),

// Load all CSV files using the function
loadedCSV = List.Transform(csvFiles, each loadCSV([Folder Path] & [Name])),
combinedData = Table.Combine(loadedCSV),

// Set data types for each column
typedData = Table.TransformColumnTypes(combinedData,
List.Transform(Table.ColumnNames(combinedData), each {_, type text}))
in
typedData
 
Back
Top