Questions tagged [pipeline]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

0
votes
1answer
5 views

HTTP persisted connection with pipeline. How multiple files are divided in segments by TCP

I have a very simple question which answer seems impossible to find as I want it. Let’s suppose I have a tcp connection between a client and a sever and the client made a request for a page that ...
1
vote
0answers
11 views

Add a feature to a pipeline with FeatureUnion

In the code below I use a tweeter dataset to perform sentiment analysis. I use a pipeline which performs the following processes so far: 1) performs some basic text preprocessing 2) vectorizes the ...
0
votes
0answers
37 views

Why can't I access /var/run/docker.sock despite being Jenkins user within docker's group?

I have Jenkins on Docker, both Jenkins master and slave are Docker containers, with mounted var/run/docker.sock. There is proper connection between them, but when I try to run simple docker "hello-...
0
votes
0answers
7 views

Unable to create directory '{p}'. Unable to verify the directory exists: '{testDir}'

Publishing build artifacts failed with an error: Unable to create directory '{p}'. Unable to verify the directory exists: '{testDir}'. If directory is a file share, please verify the share name is ...
0
votes
1answer
21 views

What are these comands mean in buildkite?

What do these command mean in buildkit build pipeline? command: command: | command: >- I am trying to build a build pipeline and I can't find any documentations on them. What are the differences ...
0
votes
1answer
32 views

how to get the pipeline name dynamically in ADF?

I am trying to implement mail notification in pipeline in ADF. So basically I want to send the a notification mail about the successful execution of pipeline in ADF. I have created a demo pipeline ...
0
votes
0answers
10 views

How to stop a detached step in drone.io?

In drone you can detach steps as seen here: https://docs.drone.io/config/pipeline/steps/ Example use case: I started a detached database. Some tests run against it. Then the db is no longer needed so ...
-1
votes
1answer
22 views

Is it possible to connect between TFS Azure boards to its pipeline continuous integration?

I would like to connect my current azure sprint boards with different work items and tasks to azure pipeline release automation. When mail sents to people asking for approval, I want to have more ...
-1
votes
0answers
15 views

too many values to unpack (expected 2) ####

I have this code and gives me the error too many values to unpack (expected 2) any suggestions? mi_pipeline2 = Pipeline(steps=[('clean_crehist', CreditHistoryTransformer(), ['credit_history']), ...
-1
votes
0answers
15 views

How to use Panel Pipeline to make a dashboard?

I am trying to make a stand-alone dashboard for GUI (the user inputs information in widgets and it graphs itself once inputted.) I do not really know how to start with the examples that are giving on ...
0
votes
1answer
25 views

Problems while reloading pickled sklearn pipeline. Countvector analyzer function not being imported

I am trying to pickle my text classification model and reload into a flask application interface. I have a specific function that I use as analyzer called as split_into_lemmas def split_into_lemmas(...
-2
votes
0answers
12 views

Jenkins pipeline for Git grade and push

I am working on gitops work for that I am building CI pipeline which includes below steps. Pull branch dev Build using gradle Pass code to Sonar Qube Check quality gate If positive then push that ...
0
votes
1answer
28 views

AWS data pipeline, run Unload of redshift from bash shell script and save to S3

I like to run unload to get data from Redshift to s3, can I run it using bash shell script of Data pipeline? if yes, what is the command inside shell script? Is the following code working in shell ...
0
votes
1answer
41 views

Is it possible to download files during the build pipeline on Azure DevOps?

We're starting to use Azure DevOps to build and deploy my application. Currently, we do not upload the application images to our repo. I Would like to know if I could download all the images to the ...
0
votes
1answer
38 views

How to Codeception and Gitlab CI (Continuous Integration)

I am currently running my acceptance tests via codeception manually. Now its time to do the next step. I want to create a pipeline in Gitlab to run my acceptance tests automatically. The very first ...
0
votes
0answers
36 views

Using bagging and random forests together

I was looking at a kernel implementation (for text classification) and the following piece of code got me a little bit confused (I removed part of the features - in order to keep it light - as most of ...
0
votes
1answer
33 views

How to apply lambda function to clean column names in pandas?

I have a dataset with column names having typos say "Address %", "Unit ", I need to remove % and strip the white space Simply below will work X.columns = X.columns.str.replace('%', '') X.columns = X....
0
votes
1answer
23 views

How to use best estimator from pipeline to predict test set?

I developed a pipeline using XGBoost which returned me a best estimator. However, trying to use this best estimator to predict my test set the following error is raised: "ValueError: Specifying the ...
0
votes
0answers
12 views

How to manage scope in a diy etl pipeline in Python

How would I go about constructing my own ETL pipeline in Python? I'm aware of using composition f(g(h())), but apparently it isn’t that simple (no branching/conditionals?) or we wouldn’t have complex ...
0
votes
0answers
15 views

Scikit-learn pipeline params contains whole steps, could that cause problems?

A sklearn pipeline step appears as a whole param and it's params also appears alonside itself when using "get_params". So here is the simplest possible example of what my problem is. When I do that: ...
-1
votes
1answer
36 views

PySpark Pipeline Error when using Indexer and Encoder

I'm using the bank data from UCI to just template out a project. I was following the PySpark tutorial on on their documentation site (sorry cant find the link anymore). I keep getting an error when ...
0
votes
1answer
24 views

TeamCity - How to cancel multiple builds via REST API

Hello DevOps Evangelists! First of all, thanks to this answer, I was able to successfully cancel single TeamCity build using following curl: curl http://teamcity:8111/app/rest/buildQueue/buildType:&...
0
votes
0answers
21 views

ValueError: too many values to unpack (expected 3) when fitting model using pipelines

I am still new to this and have a lot to learn, so bear with me while I learn how to do this well. I took a Kaggle data set and wanted to work on using pipelines to make my code neater and to simplify ...
0
votes
1answer
18 views

How to get Jenkinsfile name or the path?

Want to obtain the Jenkinsfile's name or the path of it. getProtectionDomain and File classes are restricted. Have the following structure -pipeline-development --Jenkinsfile-development.groovy -...
1
vote
1answer
57 views

how to call parameters from groovy map script to Jenkins Pipeline

simply I need to write jenkinsfile pipeline which contains choices and this choices come come from another groovy script that provides a map in this map I can add environment name and 3 values DB , ...
0
votes
1answer
46 views

Directory '/Users/vsts/agent/2.152.1/work/16/a' is empty. Nothing will be added to build artifact 'drop'

I'm getting below warning while building Artifacts on Azure Deops pipeline. Unable to integrate continuous deployment because publish Artifact for Xamarin iOS project: [warning]Directory '/Users/...
0
votes
0answers
29 views

Creating pipeline in sklearn with custom functions?

How to create sklearn pipeline with custom functions? I have a two functions, one for cleaning data and second for building model. def preprocess(df): ………………. # clean data return df_clean ...
0
votes
0answers
3 views

Running pipeline starting from another pipeline result

I need to create a pipeline to do unit and integrations tests and it will executed after lint pipeline on GO CD. I've created a pipeline having as a material the previous pipeline (lint) but and the ...
0
votes
1answer
16 views

How to configure Jenkins “Pipeline script from SCM” with SVN

In my Jenkins project, I've configured "Pipeline script from SCM" with Subversion as the SCM, a Repository URL corresponding to a repository on our test VisualSVN server (for sake of discussion, call ...
0
votes
3answers
62 views

Build is failing Error forwarding the new session Empty pool of VM for setup Capabilities browserName: chrome

So I am running my tests using Jenkins, the Pipeline script is doing all for me but the build is failing at the point where it's saying: WebDriver::debugWebDriverLogs method has been called when ...
1
vote
1answer
36 views

--latency-wait Error MissingOutputException

I am working on snakemake to make a pipeline for my work. My input files are (Example fastq files) /project/ateeq/PROJECT/snakemake-example/raw_data/H667-1_R1.fastq.gz /project/ateeq/PROJECT/...
0
votes
0answers
16 views

I have a automatic build job will copy package to artifactory, deployment pipeline job how to pick the latest package version from artifactory

I have Build job using jenkins pipeline and after building it will generate war package and upload to artifactory structure like: http:///artifactory/webapp/#/artifacts/browse/tree/General/binaries///...
1
vote
0answers
36 views

How to efficiently pipe data of HTTP GET to HTTP PUT in .NET?

I would like to asynchronously download data and then upload it again using .NET. Ideally this should work in .NET and .NET Core as I need to use this in PowerShell 5 and 6 in the end. I searched for ...
0
votes
0answers
40 views

Instruction pipelining - how is store/load behaving?

So, if I have a pipeline with Fetch, Decode, Execute, Memory Access and Write Back, how does the store/load function work? For example if I have sw $1,$2 (store a word from $2 to $1) will it do ...
0
votes
0answers
8 views

How to terminate the whole data pipeline inside the shell command activity (aws)?

So in the ShellCommandActivity I run some commands to check the input file, if it is valid it will run a few next commands. If not, I want to terminate the whole pipeline. Previously I just exit ...
-1
votes
0answers
14 views

Need help using sklearn feature union to combine text and categorical data

Trying to use a sklearn pipeline and feature union to combine text data and categorical data into one model. I think I'm close. I followed another stackoverflow thread to make some of my code. But i'm ...
0
votes
1answer
58 views

Kubeflow pipeline error: failed to save outputs: No such container

While trying to setup my own kubeflow pipeline I ran into a problem when one step is finished and the outputs should be saved. After finishing the step kubeflow always throws an error with the message ...
1
vote
0answers
17 views

Apache beam python error handling while reading files from file pattern

I am reading all files from a directory and writing to the bigquery table. If there is an error with any file in the directory, it will raise the error and stop the job. I am not getting any ...
0
votes
1answer
34 views

How can I access pipeline parameters from data flow?

I have a pipeline with a parameter (type int). Inside the pipeline, at one point, I have a data flow. This data flow needs to reference this parameter in order to filter data by it and add it as a new ...
0
votes
1answer
34 views

Error when converting bash code to snakefile

I want to generate genome index by using STAR. The bash code works in the terminal but I want to convert it to snakefile. This is the bash code: STAR --runThreadN 4 --runMode genomeGenerate --...
0
votes
1answer
37 views

Why modern processors still use in-order pipeline?

I was searching about the ARM Cortex-A53 processor and i found out it uses a static in-order pipeline, in that instructions issue, execute, and commit in order. I can't understand why a modern ...
0
votes
0answers
19 views

Bitbucket Pipeline YAML for loop script syntax error

I'm trying to use the AWS CLI to invalidate my CloudFront distribution. This consists of two commands, one to create the invalidation and then one to wait for it to complete. To do this, I need to ...
0
votes
2answers
60 views

Making a bullet-proof Git checkout in our CD job

The simplified story is that we have a site installed on multiple servers (e.g QA, Prod). Git is set to another branch/tag for every environment, and we have deployment scripts that are generic for ...
0
votes
0answers
9 views

Can we have the jenkins shared library within jenkins dsl code base?

I have a jenkins shared library and dsl code base. I would like to use few classes/methods from the dsl codebase in the shared library. Is there anyway to do this? I tried copying the shared ...
0
votes
1answer
19 views

scrapy-elasticsearch pipeline only for specific item

I want to use the scrapy-elasticsearch pipeline in my scrapy project. In this project I have different items / models. These items are stored in a mysql server. In addition I want to index ONE of ...
0
votes
0answers
10 views

How to pass 4D Array to scikit-learn pipeline without raising ValueError: Found input variables with inconsistent numbers of samples

I have a list of filepaths to 3D Nifti Files. I use nilearn’s resample_img which returns a 4D Image, where the first three dimensions represent x,y,z axes and the fourth dimension represents number of ...
0
votes
3answers
87 views

Is there any way to tell apart multiple multi-line outputs from a script?

I am running a script input.sh which has multiple multi-line outputs like so: echo -e 'first \n second' echo -e 'first \n second' echo -e 'first \n second' I don't have control over this file, and ...
1
vote
1answer
34 views

Vulkan: one pipeline and multiple descriptor sets?

I'm trying to create a single pipeline with a layout where it requires two bindings, a dynamic UBO and a image/sampler binding. I want each binding to come from a separate descriptor set, so I'd bind ...
0
votes
0answers
21 views

how to pipe in h264 frame byte array into ffmpeg and mux intro rtmp c#

x264 -> h264 frame byte[] array -> pipe -> ffmpeg -> rtmp only reason im not using ffmpeg as the encoder is need direct parameter update with encoder every second proc.StartInfo.Arguments = String....
0
votes
1answer
19 views

GOCD api to compare changes between two triggers in pipelines

I want to get all the changes between two builds in a pipeline in GOCD api. I read the api documentation of GOCD. Couldn't find any api. Like this url from gocd dashboard -> https://localhost:8153/...